Fixed: "Server shutdown in progress" in EF Core with MySQL
MySQL was restarting, and a statement in the middle of running failed with MySqlException: Server shutdown in progress. That's error 1053, it's temporary, and the server will be back in seconds. The fix is one number in the retry list.
What I Was Doing
My .NET C# Web API uses EF Core 10.0.14 and MySql.Data 26.10.0. An ExecuteUpdateAsync was running while MySQL restarted:
docker compose restart mysql
What I Expected
A short wait, and then an answer.
What I Got
500 1.106149s
MySql.Data.MySqlClient.MySqlException (0x80004005): Server shutdown in progress
What I Checked
The number. MySQL's server error reference lists it as error 1053, ER_SERVER_SHUTDOWN, with the message Server shutdown in progress.
Whether it was retried. No: it wasn't in my errorNumbersToAdd, and Oracle's provider retries nothing else.
Why It Happens
When MySQL stops, it ends the statements that are running, and this is what it tells their connections. If the connection is cut before the message arrives, you get Reading from the stream has failed instead, so a restart can produce either one.
The Fix
Add 1053 to the errors you retry:
protected override bool ShouldRetryOn(Exception exception) =>
exception is MySqlException { Number: 0, InnerException: not null }
or MySqlException
{
Number:
1040 or // Too many connections.
1042 or // Unable to connect to any of the specified MySQL hosts.
1053 or // Server shutdown in progress.
1213 // Deadlock found: MySQL rolled the transaction back.
};
With EnableRetryOnFailure, that's errorNumbersToAdd: [1053], along with whichever others you need.
After the Fix
{"changed":1}
curl exit 0, 5.502409 s, status 200
A transient exception occurred during execution. The operation will be retried after 0ms.
A transient exception occurred during execution. The operation will be retried after 1044ms.
A transient exception occurred during execution. The operation will be retried after 3273ms.
It took 5.5 seconds, and it answered 200. The client just saw a slow request.
Common Mistakes
| Mistake | What happens | Do this instead |
|---|---|---|
Listing only 1053 | The next attempts fail differently while MySQL is down | List 0 and 1042 too |
| Retrying non-repeatable work | A repeat can apply it twice | Add a key |
| Too few retries | The restart outlasts them | Answer 503 when they run out |
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Unable to connect to any of the specified MySQL hosts | MySQL is down now | The fix |
NullReferenceException during the restart | A retry on a broken connection | The fix |
The maximum number of retries (3) was exceeded | The restart took longer than the retries | The fix |
Conclusion
Server shutdown in progress is error 1053: MySQL is stopping, and it'll be back. Retry it, with the other errors a restart causes.
Additional Resources
- Server Error Message Reference — error 1053.
- Connection Resiliency — retries in EF Core.
