I am not entirely sure whether this issue is a CrateDB issue or an issue with the EFCore provider I am using, but I figure I will start here, and if you cannot help me I will try wth the provider.
So, I recently started using CrateDB, but while setting up a test database to examine its performance, I ran into an issue that happens when saving data; in short I frequently - but not always - get an exception thrown by my DbContext when I try to save batches of data.
My setup is as follows: I am using Entity Framework Core, version 8.0.22, with Npgsql.EntityFrameworkCore.PostgreSQL 8.0.11 as my database provider.
I made a DbContext containing a single table with a fairly basic schema, pictured here:
I have verified that I can save and retrieve records just fine with a basic DbContext under this setup.
However, when saving rows in bulk (350+ rows at once), I sometimes get the following exception:
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
---> Npgsql.NpgsqlException (0x80004005): Received backend message CommandComplete while expecting ParseCompleteMessage. Please file a bug.
at Npgsql.Util.Statics.ThrowIfMsgWrongType[T](IBackendMessage msg, NpgsqlConnector connector)
at Npgsql.Util.Statics.Expect[T](IBackendMessage msg, NpgsqlConnector connector)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at DatabaseComparisonExperiment.Program.SaveTest(TestDbContext dbContext) in C:\Users\jbc\SynologyDrive\svn\OptiCloudsAPI\DatabaseComparisonExperiment\Program.cs:line 248
Sometimes, I do not get this exception, sometimes I do get it. Sometimes I get the exception, but all rows saved successfully anyway, sometimes there are rows missing.
If I ignore this exception and continue with the next batch, sometimes the next batch saves just fine, other times the issue repeats, even if the batch is less than 100 rows.
My DbContext is configured as follows:
public class TestDbContext(string connectionString) : DbContext
{
protected readonly string ConnectionString = connectionString;
public virtual DbSet<SaveTester> SaveTesters { get; protected set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseNpgsql(ConnectionString);
base.OnConfiguring(optionsBuilder);
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<SaveTester>(entity =>
{
entity.ToTable("SaveTest");
entity.HasKey(x => new { x.Timestamp, x.Node });
entity.Property(x => x.Value).IsRequired();
});
base.OnModelCreating(modelBuilder);
}
}
The code that I am using to test the database is as follows:
DateTime timestamp = DateTime.UtcNow.AddYears(-1);
for (int batchSize = 350; batchSize < int.MaxValue; batchSize += 10)
{
Console.WriteLine($"Testing batch size of {batchSize}...");
List<SaveTester> saveTesters = [];
for (int ix = 0; ix < batchSize; ix++)
{
timestamp = timestamp + TimeSpan.FromSeconds(0.1 + (Random.Shared.NextSingle() * 0.4)); //Random interval between datapoints of 0.1 - 0.5 seconds, mimicking real world conditions.
saveTesters.Add(new(time: timestamp, node: "TestNode", value: Random.Shared.NextSingle()));
}
dbContext.SaveTesters.AddRange(saveTesters);
try
{
await dbContext.SaveChangesAsync();
Console.WriteLine("Success.");
}
catch (Exception ex)
{
Console.WriteLine($"Failure due to exception:\n{ex}");
if(!saveTesters.Any(x => dbContext.SaveTesters.Contains(x))
Console.WriteLine("However, all rows saved successfully anyway");
batchSize /= 2;
}
}
EDIT: I should add that the issue occurs both when I use the asynchronous dbContext.SaveChangesAsync() and the synchronous dbContext.SaveChanges().
I will add that my issue seems similar to this issue for EF5:
But given the version difference and the fact that the issue was marked as solved, I consider this a new issue.
Any help would of course be appreciated, though I am writing this as I am on my way out the door and may not respond until tomorrow.