Hello,
I’m running CrateDB version 4.6.3 and I’ve read the following:
- COPY TO — CrateDB: Reference
- Export table using COPY TO command to alternate S3 endpoint - #2 by miguel.arregui
but I still couldn’t manage to export data to our S3-compatible object storage running in our data center (on-premise). It seems like CrateDB is trying to contact AWS endpoint, but I need it to contact our on-premise object storage.
I tried the following:
COPY emre.test_table_002 TO DIRECTORY 's3://access_key:secret_key@our.objectstorage.cloud/bucket001-benchmark/' ;
and received:
NotSerializableExceptionWrapper[amazon_s3_exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: K7FS0STJ2C38XGKQ; S3 Extended Request ID: oEir5xjpqyh+xGAtbAjjuSM+lUUlrRV75U2BSgCPQbh2e+NBnSkw8kourpXicD2xiiXgJPvpJuY=; Proxy: null)]
org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: amazon_s3_exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: K7FS0STJ2C38XGKQ; S3 Extended Request ID: oEir5xjpqyh+xGAtbAjjuSM+lUUlrRV75U2BSgCPQbh2e+NBnSkw8kourpXicD2xiiXgJPvpJuY=; Proxy: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
...
That object storage is working, e.g. using s5cmd I can interact with it such as:
$ s5cmd --no-verify-ssl --endpoint-url https://our.objectstorage.cloud ls s3://bucket001-benchmark
2020/06/02 08:39:20 1771503 file_01.pdf
2021/03/19 09:55:09 14149579439 file_02.tar.gz
And I can copy files to that object storage, again using s5cmd like:
$ s5cmd --no-verify-ssl --endpoint-url https://our.objectstorage.cloud cp file03.tar.gz s3://bucket001-benchmark/
How can I make CrateDB use our S3-compatible object storage instead of AWS?