Issue with adding new fields in the object data type

CrateDB
1

In my crate setup, I noticed below type of logs.

Sep 10 16:40:00 cp-20-44-50-229 crate[257638]: [2024-09-10T16:40:00,296][WARN ][o.e.i.s.IndexShard ] [cp-20-44-50-229.internal] [test..partitioned.t1.04732dpi6ksj4dhk60o30c1g][8]no index mapper found for field: [dataobj.place] returning default postings format

This is because my java program was trying to insert a new column in the dataobj ( type object) column of table. Table was created using below statement.

CREATE TABLE "doc"."t2" (
   "id" BIGINT NOT NULL,
   "idx" BIGINT,   
   "id2" BIGINT NOT NULL,
   "text3" TEXT,
   "text2" TEXT,   
   "birthday" TIMESTAMP WITH TIME ZONE NOT NULL,
   "text4" TEXT,
   "dataobj" OBJECT AS (
      "countin" BIGINT,
      "countout" BIGINT
    ),    
   PRIMARY KEY ("id", "id2", "birthday")
)PARTITIONED BY ("birthday")
WITH (
   "translog.durability" = 'ASYNC'
);

Only change I had done was to increase the number of shards for this table and I believe after that was done, this warn message is appearing and new rows are not getting inserted.

I tried to reproduce it by creating a new table with same schema and then ran below commands.

insert into doc.t2 ( "id", "id2", "birthday", dataobj) values (1,1,1725129000000, '{"city":"LA"}') ;

This was successful. Then I ran below command to increase shards.

ALTER table doc.t2 set (number_of_shards = 6);

Then ran below command to insert second row with new field in dataobj column.

insert into doc.t2 ( "id", "id2", "birthday", dataobj) values (2,1,1725215400000, '{"city":"LA","place":"hollywood"}') ;

This did not get inserted. But I did not get the same error as I got in first case. But data is not getting inserted.

Is there any reported bug in crate or fix available in newer versions. I am using crate 5.6.4. Kindly help.

2

When altering a partitioned table things are a bit more complex, see https://cratedb.com/docs/crate/reference/en/latest/general/ddl/partitioned-tables.html#changing-the-number-of-shards

I was able to reproduce your case in 5.6.4 it leaves the table in an illegal state, in 5.7.4 I could not reproduce it and works fine, could you try updating your cluster to 5.7.5 or the latest 5.8.*

1 Like
3

Hi Vinayak Shukre,

As Iván said, this bug is already fixed in versions 5.7.5 or 5.8.2, you can see the original bug here: Cannot add new column to a partitioned table after changing the number of shards · Issue #16521 · crate/crate · GitHub

1 Like
4

Thank you. I will try this.

I am checking here and it suggests below.

We recommend that you upgrade to the latest 5.7 release before moving to 5.8.2.

Should I upgrade all nodes of 5.6.4 to 5.7.5 and then start full cluster, ensure that it is green and then upgrade to 5.8.2?
Or is it only upgrade to 5.7.5 and then immediately to 5.8.4 and then start cluster? Kindly guide.