Fixed: "NullReferenceException" at NativeDriver.ExecutePacketAsync During a MySQL Restart in EF Core
MySQL restarted while orders were coming in. The retries did their job for most requests, and then 8 of 96 failed with System.NullReferenceException from deep inside MySql.Data. None of my code was in the failing frames. A retry was using a connection that the driver had left marked as open.
What I Was Doing
My .NET C# Web API uses EF Core 10.0.14, MySql.EntityFrameworkCore 10.0.12 and MySql.Data 26.10.0, with retries turned on for the errors a restart causes:
mySql.EnableRetryOnFailure(
maxRetryCount: database.MaxRetryCount,
maxRetryDelay: TimeSpan.FromSeconds(database.MaxRetryDelaySeconds),
errorNumbersToAdd: [0, 1040, 1042, 1053, 1213]);
Orders were arriving, 8 at a time, 96 in all:
for round in 1 2 3 4 5 6 7 8 9 10 11 12; do
seq 8 | xargs -P 8 -I{} curl -s -o /dev/null -w "%{http_code}\n" -X POST http://localhost:5190/api/v1/orders -H "Content-Type: application/json" -d '{"customerName":"Buyer {}","customerEmail":"buyer{}@example.com","lines":[{"productId":1,"quantity":1}]}'
sleep 0.5
done | sort | uniq -c
Meanwhile, MySQL restarted:
docker compose restart mysql
What I Expected
Every order would wait through the restart, and then succeed.
What I Got
88 201
8 500
System.NullReferenceException: Object reference not set to an instance of an object.
at MySql.Data.MySqlClient.NativeDriver.ExecutePacketAsync(MySqlPacket packetToExecute)
What I Checked
Whether it was bad luck. The next run gave exactly the same split:
88 201
8 500
Where the exception starts. The log has the retries first, and the failures after them. Each failure is the first query of a retried attempt, going down through ExecutionStrategy into the driver:
An exception occurred while iterating over the results of a query for context type 'Dewiride.Inventory.Data.InventoryDbContext'.
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
An exception occurred while iterating over the results of a query for context type 'Dewiride.Inventory.Data.InventoryDbContext'.
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
So it happens on a retry, on the first statement that uses a connection that's already in trouble.
The driver's source. In MySqlConnection.cs, opening a connection marks it as open, and only then selects the database:
SetState(ConnectionState.Open, false);
await driver.ConfigureAsync(this, cancellationToken).ConfigureAwait(false);
// ...
if (!string.IsNullOrEmpty(Settings.Database))
await ChangeDatabaseAsync(Settings.Database, cancellationToken).ConfigureAwait(false);
Why It Happens
From the source and the stack, here's how I read it. During a restart, a pooled connection is dead, and the driver doesn't know yet. Opening it reaches ChangeDatabaseAsync, which fails. That's the error EF Core sees, and retries. But the connection object is still marked Open, and it can no longer talk to MySQL. On the retry, EF Core sees an open connection, and doesn't open it again. So the next statement, whatever it is, runs on it, and the driver hits a null inside ExecutePacketAsync. When an attempt starts with BeginTransactionAsync, the same stack goes through that instead.
The NullReferenceException isn't on the retry list, and it shouldn't be, so that request fails.
The Fix
Close the connection before each retry. EnableRetryOnFailure has no place for that, but ExecutionStrategy has a method for it, OnRetry, so a small class of your own does the job:
using System.Data;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Storage;
using MySql.Data.MySqlClient;
namespace Dewiride.Inventory.Data;
/// <summary>
/// Decides which MySQL errors are worth repeating an operation for. Oracle's provider has its own
/// strategy, but it treats no MySQL error as temporary unless it's listed, and it can retry on a
/// connection the driver has left broken. This one lists the errors and starts each retry clean.
/// </summary>
public sealed class MySqlRetryStrategy(
ExecutionStrategyDependencies dependencies,
int maxRetryCount,
TimeSpan maxRetryDelay)
: ExecutionStrategy(dependencies, maxRetryCount, maxRetryDelay)
{
// Number 0 is the driver's own error. It's a dropped connection when an inner exception says
// why, and a full connection pool when there's none: a retry would only wait for the pool again.
protected override bool ShouldRetryOn(Exception exception) =>
exception is MySqlException { Number: 0, InnerException: not null }
or MySqlException
{
Number:
1040 or // Too many connections.
1042 or // Unable to connect to any of the specified MySQL hosts.
1053 or // Server shutdown in progress.
1213 // Deadlock found: MySQL rolled the transaction back.
};
protected override void OnRetry()
{
// When a connection fails while it's being opened, MySql.Data can leave it marked as open.
// The next attempt would then use it and fail with a NullReferenceException.
var connection = Dependencies.CurrentContext.Context.Database.GetDbConnection();
if (connection.State != ConnectionState.Closed)
{
connection.Close();
}
}
}
Use it in place of EnableRetryOnFailure:
mySql.ExecutionStrategy(dependencies => new MySqlRetryStrategy(
dependencies,
database.MaxRetryCount,
TimeSpan.FromSeconds(database.MaxRetryDelaySeconds)));
How to Write a Custom Execution Strategy in EF Core for MySQL goes through it line by line.
After the Fix
96 201
All 96 got through, with 27 retries in the log and no NullReferenceException.
Common Mistakes
| Mistake | What happens | Do this instead |
|---|---|---|
Adding NullReferenceException to what's retried | Every real bug in your code gets retried | Close the connection before retrying |
| Clearing the pool on every failure | Healthy connections go too, and it doesn't fix this one | OnRetry closes only this one |
| Testing retries with MySQL stopped | This only shows with stale pooled connections | Restart MySQL while requests run |
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
The stack ends in MySqlCommand.ExecuteNonQueryInternalAsync | A different problem | KILL QUERY |
Reading from the stream has failed, with no retries | The connection dropped, and 0 isn't listed | The fix |
| Requests saved twice after the restart | A retry repeated a whole transaction | Clear the tracker |
FAQ
Is this a bug in MySql.Data?
It looks like one to me: a connection that failed to open shouldn't report itself as open. I tested MySql.Data 26.10.0, so check again after an upgrade.
Conclusion
A retry can pick up a connection that MySql.Data left marked as open after it failed to open it, and that ends in a NullReferenceException. Close the connection in OnRetry, in a small ExecutionStrategy of your own, and a restart under load stops costing you requests.
Additional Resources
- Connection Resiliency — execution strategies in EF Core.
- MySqlConnection.cs — where the connection is marked as open.
