Using CrateDB Toolkit to sync MongoDB with CrateDB

CrateDB
21

Hello , tks you for your update and I upgraded version to 0.0.38 , but when I tried running command , it doesn’t work although I waited 1 hours and don’t have anything happen , it’s still

(ctk-env) ubuntu@ip-172-31-19-57:\~/ctk-env$ export CRATEDB_CLUSTER_URL=“crate://u:p@h/cloudstore/sale?ssl=true”
(ctk-env) ubuntu@ip-172-31-19-57:\~/ctk-env$ ctk load table “mongodb://u:p@h/CloudStore/Sale”
2025-08-14 03:09:39,801 \[cratedb_toolkit.io.mongodb.api      \] INFO    : mongodb_copy. source=mongodb://u:p@h/CloudStore/Sale, target=crate://u:p@h/cloudstore/sale?ssl=true
22

Hi,

this looks like the connection might stall, so the process likely doesn’t even start to transfer data?

Let’s see if I can provide you with a basic full end-to-end walkthrough. Then, we can validate if that one works on your end. Maybe this helps to find the root cause.

With kind regards,
Andreas.

23

yah , thanks you . Do you mind if we can have a live demo to solve this problem ? Or you can guide me step-by-step in any version , I can follow it ^^ . After tried fail with version 0.0.38 , I tried to run ctk 0.0.36 with pip and your format command but not work .

(ctk-env) ubuntu@ip-172-31-19-57:~/ctk-env$ ctk --version
ctk, version 0.0.36
(ctk-env) ubuntu@ip-172-31-19-57:~/ctk-env$ ctk load table   "mongodb://root:pass@host:port/CloudStore?table=CloudStore.Authentication&authSource=admin&readPreference=secondary"   --cluster-url="crate://system:pass@host/cloudstore/authentication?ssl=true"
2025-08-14 07:00:49,071 [cratedb_toolkit.io.mongodb.api      ] INFO    : mongodb_copy. source=mongodb://root:pass@host:port/CloudStore?table=CloudStore.Authentication&authSource=admin&readPreference=secondary, target=crate://system:pass@host/cloudstore/authentication?ssl=true
2025-08-14 07:00:49,075 [cratedb_toolkit.io.mongodb.api      ] INFO    : Inquiring collections at mongodb://root:pass@host:port/CloudStore?table=CloudStore.Authentication&authSource=admin&readPreference=secondary
/home/ubuntu/ctk-env/lib/python3.12/site-packages/pymongo/uri_parser.py:329: UserWarning: Unknown option: table. Did you mean one of (tlscafile, tls, tz_aware) or maybe a camelCase version of one? Refer to docstring.
  return get_validated_options(opts, warn)
2025-08-14 07:00:49,495 [cratedb_toolkit.io.mongodb.api      ] INFO    : Discovered collections: 92
Traceback (most recent call last):
  File "/home/ubuntu/ctk-env/bin/ctk", line 7, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/cratedb_toolkit/io/cli.py", line 69, in load_table
    cluster.load_table(source=source, target=target, transformation=transformation)
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/cratedb_toolkit/cluster/core.py", line 587, in load_table
    if mongodb_copy(
       ^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/cratedb_toolkit/io/mongodb/api.py", line 170, in mongodb_copy
    MongoDBFullLoad(
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/cratedb_toolkit/io/mongodb/copy.py", line 43, in __init__
    self.cratedb_sqlalchemy_url, self.cratedb_table_address = self.cratedb_address.decode()
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/cratedb_toolkit/model.py", line 153, in decode
    database, table = decode_database_table(self.dburi)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ctk-env/lib/python3.12/site-packages/cratedb_toolkit/util/database.py", line 472, in decode_database_table
    raise ValueError("Database and table must be specified")
ValueError: Database and table must be specified
24

Excellent.

Now you only have to get rid of the URL syntax suitable for ingestr, and use the other URL syntax instead. It is mongodb://<host>/<database>/<collection> and crate://<host>/<schema>/<table>, no table parameter in sight.

2 Likes
28

it’s working . Thanks you very much <3

1 Like