Problem using the COPY FROM from AWS S3

CrateDB
1

Hello all,

(using version: 5.10.16 on 3 nodes)

I’ve been using AWS S3 buckets recently to transfer data between tables for reindexing/resharding purposes. We used COPY TO/FROM in bash script with crash for this , which has worked very well.

We’re doing one specific table at the moment where COPY TO has worked fine, but suddenly when using COPY FROM we are gettingS3 errors indicating that it cannot find the SOME of the files.

There are 224 json files in the folder and it seems to be skipping 50 of them.
when retrying and even doing each individual file in the webconsole is causing the same issue:

CONNECT OK AmazonS3Exception[The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: KRNDK9H9QFE5Y050; S3 Extended Request ID: PsWaG0OTr84Er3I62aVNAsZC5kX7I3QPW0BMhVCTJ51dZr7AS4M1lF4T6fnxKHnq80zDJkRZJgjLlScFJsSIn548BYsBZdfh; Proxy: null)]

I need to point out that I restore the gz or json files individually and not through file globbing e.g. /*.gz (/*.json) as that does not find any files!

All the other files in the same bucket process fine.

Many thanks in advance
David

2

It sounds like a path problem. check these:

Case sensitivity: s3 is picky about file names, so make sure the case matches exactly.

Full file path: make sure you use the whole path, which includes the folder and the file name.

File visibility: it can take s3 a minute to update. Wait a little while before trying again.

Permissions: make sure your IAM role can read all the files, especially the ones it’s missing.

You could also try it out with just a few files to see if it’s a one-time problem or something bigger.

1 Like