Fixes for Common Issues and Errors
View All Tags
MySQL restarted while orders were coming in. The retries did their job for most requests, and then 8 of 96 failed with System.NullReferenceException from deep inside MySql.Data. None of my code was in the failing frames. A retry was using a connection that the driver had left marked as open.
A request took about 4 seconds, and then failed with RetryLimitExceededException: The maximum number of retries (3) was exceeded while executing database operations. That exception means EF Core did its job and ran out of attempts. The cause is one level down, and the client deserves a better answer than a 500.
An order hung for a while and then failed with MySqlException: Lock wait timeout exceeded; try restarting transaction, and the client got a bare 500. Another transaction held the row. Three things make this better: a shorter wait, a clearer answer for the client, and finding what's holding the row.
Under load, half the orders failed with MySqlException: Deadlock found when trying to get lock; try restarting transaction, and with an HTTP 500. The code looked fine. Each order just locked its products in whatever order the client listed them, so two orders could each hold what the other needed.
Wrapping an endpoint's work in strategy.ExecuteAsync gave error CS8031: Async lambda expression converted to a 'Task' returning delegate cannot return a value. The code looks right, and it returns values on purpose. The compiler just can't work out one type for those values, and a small addition tells it.
As soon as I wrapped two writes in BeginTransactionAsync, the endpoint failed with The configured execution strategy 'MySQLRetryingExecutionStrategy' does not support user-initiated transactions. The message tells you the method to use, and here's the shape of the code it means.
A request that normally takes a millisecond failed after exactly 15 seconds with error connecting: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. MySQL itself was fine and nearly idle. All the app's connections were taken, by requests that were never going to finish.
I set a short CommandTimeout, expecting an error when a statement ran too long. Instead, the request never answered at all. MySQL had finished long ago, and the connection sat idle, out of reach of everything else. If you're on Oracle's MySql.Data, any async statement slower than CommandTimeout can do this, including a default of 30 seconds behind a row lock.
I had EnableRetryOnFailure turned on, so I assumed a deadlock or a short outage would be retried. With Oracle's MySQL provider, nothing was retried, ever, and no message says so. The provider asks each error whether it's temporary, and MySqlException always answers no. The fix is one argument.
SET SESSION innodb_print_all_deadlocks = ON failed with ERROR 1229 (HY000): Variable 'innodb_print_all_deadlocks' is a GLOBAL variable and should be set with SET GLOBAL. The message tells you exactly what to do. There's also a catch, which is that it takes an administrator.
