Fixed: "Fatal error encountered during command execution" and "Reading from the stream has failed" with MySql.Data
Fatal error encountered during command execution sounds like something is badly wrong with the statement. Read the inner exceptions, and it ends with Attempted to read past the end of the stream: the connection went away while the driver was waiting for an answer. The statement isn't the problem, and it's usually worth trying again.
What I Was Doing
My .NET C# Web API uses EF Core 10.0.14 and MySql.Data 26.10.0. An ExecuteUpdateAsync was running when its connection ended, from a KILL in another session. MySQL restarting does the same thing.
What I Expected
An error saying the connection had been lost.
What I Got
curl exit 0, 1.696461 s, status 500
MySql.Data.MySqlClient.MySqlException (0x80004005): Fatal error encountered during command execution
---> MySql.Data.MySqlClient.MySqlException (0x80004005): Fatal error encountered attempting to read the resultset
---> MySql.Data.MySqlClient.MySqlException (0x80004005): Reading from the stream has failed
---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
What I Checked
The chain, from the bottom up. The last line is the actual event: the driver tried to read, and the connection had ended. Each line above it is another layer of the driver adding its own wrapper.
The error number. The driver's source shows where the top message comes from:
if (mySqlException.Number == 0)
throw new MySqlException(Resources.FatalErrorDuringExecute, mySqlException);
Number 0 means the error came from the driver, not from MySQL, so there's nothing to look up in MySQL's error list.
Whether the retry saw it. With errorNumbersToAdd: [], no. Oracle's provider retries no MySQL error by itself.
Why It Happens
The driver sent a statement and was waiting for the answer when the connection ended. It could be a restart, KILL, a network failure, or a timeout on something in between. The driver can't tell you which, and it can't tell whether the statement ran, either.
The Fix
Retry it, because it's a temporary failure. With EnableRetryOnFailure, add 0 to the list. A custom strategy can be more exact, since a full connection pool is also number 0, and that one isn't worth retrying:
// Number 0 is the driver's own error. It's a dropped connection when an inner exception says
// why, and a full connection pool when there's none: a retry would only wait for the pool again.
protected override bool ShouldRetryOn(Exception exception) =>
exception is MySqlException { Number: 0, InnerException: not null }
or MySqlException
{
Number:
1040 or // Too many connections.
1042 or // Unable to connect to any of the specified MySQL hosts.
1053 or // Server shutdown in progress.
1213 // Deadlock found: MySQL rolled the transaction back.
};
Then make sure a repeat is safe. If the connection dropped during a COMMIT, MySQL may have committed, and a retry would do it all again.
After the Fix
{"changed":1}
curl exit 0, 3.599492 s, status 200
A transient exception occurred during execution. The operation will be retried after 0ms.
Common Mistakes
| Mistake | What happens | Do this instead |
|---|---|---|
| Reading only the top message | It looks like a bad statement | Read down to the last inner exception |
| Looking up error 0 in MySQL's list | It isn't there | It's the driver's own |
| Retrying with no way to detect a repeat | A lost COMMIT applies the work twice | Add a key |
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Server shutdown in progress | MySQL was stopping | The fix |
NullReferenceException on the retry | The connection was left marked as open | The fix |
| It happens to idle connections after a while | A network device or wait_timeout ended them | Check both |
Conclusion
These messages mean the connection dropped while the driver waited for an answer. It's number 0, so list it, or use a small strategy of your own, and make sure that repeating the work is safe.
Additional Resources
- Connection Resiliency — retries, and commit failures.
- MySqlCommand.cs — where the top message comes from.
