COPY TO fails ~ possibly file size limit reached?

CrateDB
1

Hey there,

I could not find an issue with the similar fault behaviour.

So I am using a crate cluster with 2 data nodes and 1 arbiter node, all running in docker containers. I want to copy data from some tables using the COPY TO statement to my local file system.
It works fine for small tables: All data is stored in my files (4 shards → 4 files).
Yet, for bigger tables, the process fails.

COPY ****
WHERE ***
TO directory '*****' with (compression='gzip');

The error message is not very helpful as it gives

Fehler!

[object Object]

I can see that in fact some data has been copied to my file system as I got 4 files with approx. 5 MB each… So I was wondering if this has to do with some limitation of the COPY TO process related to the file size?

I’d be happy if anyone had any suggestion - thanks a lot :slight_smile:

2

Hi @sblechmann,

can you share the CrateDB version this is happening on, please? There used to be a bug that [object Object] was shown instead of the proper error message in an older version.

3

hey @hammerhead
sure thing, it’s on version 4.6.6

4

I can’t find the exact version number right now when the issue with the error message was addressed, but there were several reports of it around CrateDB 4.6 versions.

Is upgrading to a more recent CrateDB version possible for you? The later the better, but CrateDB 4.8.4 should be enough for this particular issue.

Alternatively, do you get a proper error message when trying to run the COPY TO through another tool, such as crash. It might be only the Admin UI being affected by that bug, obfuscating the error message. You might also check if you can see the error in sys.jobs_log: SELECT * FROM sys.jobs_log WHERE error IS NOT NULL.

5

Hey @hammerhead, wow… it’s been quite some time since I took a look here, I deeply apologize!
I just updated to 4.8.4, the error and the error message are the same … yet, now each file is about 112M big.
With the sys.jobs_log: SELECT * FROM sys.jobs_log WHERE error IS NOT NULL I do not see anything.

I’ll have a look at this crash.