How to export data using `COPY TO` to non-Amazon object storage?

CrateDB
1

Hello,

I’m running CrateDB version 4.6.3 and I’ve read the following:

but I still couldn’t manage to export data to our S3-compatible object storage running in our data center (on-premise). It seems like CrateDB is trying to contact AWS endpoint, but I need it to contact our on-premise object storage.

I tried the following:

COPY emre.test_table_002 TO DIRECTORY 's3://access_key:secret_key@our.objectstorage.cloud/bucket001-benchmark/' ;

and received:

NotSerializableExceptionWrapper[amazon_s3_exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: K7FS0STJ2C38XGKQ; S3 Extended Request ID: oEir5xjpqyh+xGAtbAjjuSM+lUUlrRV75U2BSgCPQbh2e+NBnSkw8kourpXicD2xiiXgJPvpJuY=; Proxy: null)]

org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: amazon_s3_exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: K7FS0STJ2C38XGKQ; S3 Extended Request ID: oEir5xjpqyh+xGAtbAjjuSM+lUUlrRV75U2BSgCPQbh2e+NBnSkw8kourpXicD2xiiXgJPvpJuY=; Proxy: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
...

That object storage is working, e.g. using s5cmd I can interact with it such as:

$ s5cmd --no-verify-ssl --endpoint-url https://our.objectstorage.cloud ls s3://bucket001-benchmark
2020/06/02 08:39:20          1771503 file_01.pdf
2021/03/19 09:55:09      14149579439 file_02.tar.gz

And I can copy files to that object storage, again using s5cmd like:

$ s5cmd --no-verify-ssl --endpoint-url https://our.objectstorage.cloud cp file03.tar.gz s3://bucket001-benchmark/

How can I make CrateDB use our S3-compatible object storage instead of AWS?

2

Hi @Emre_Sevinc,

currently COPY TO/FROM is limited to AWS S3 and does not support S3-compatible options.

We have an open issue to support S3-compatiable alternatives (Support setting a different endpoint for COPY TO / FROM for DNS-style S3 bucket adresses · Issue #11654 · crate/crate · GitHub). This is planned for the CrateDB 4.8 release.

3 Likes
3

Actually this is already worked on, though I don’t want to make any promises when this will be released :slight_smile: