Fixed: "Unhandled exception. System.AggregateException" with "Authentication to host failed. (I/O error occurred.)" Crashes a .NET C# API with MySql.Data

No error page and no 500: the whole API process ended, in the middle of a burst of requests. The message mentions authentication and I/O, and neither was the problem. The real cause is more requests at once than Maximum Pool Size, and a callback inside the driver that throws where nothing can catch it.

What I Was Doing

I was testing connection pool sizes on a .NET 10 Web API that uses EF Core 10.0.14 and MySql.Data 26.10.0. The connection string had the default Maximum Pool Size of 100. I restarted the API, then sent 600 requests, 150 at a time:

Terminal

seq 600 | xargs -P 150 -I{} curl -s -o /dev/null -w "%{http_code}\n" "http://localhost:5190/api/v1/products?pageSize=100" | sort | uniq -c

What I Expected

A slower answer for some requests, and at worst an error for a few of them.

What I Got

Output

 587 000
   1 500
  12 503

000 means curl got no answer at all. The API's terminal showed why:

API terminal

Unhandled exception. System.AggregateException: One or more errors occurred. (One or more errors occurred. (Authentication to host '127.0.0.1' failed. (I/O error occurred.)))
 ---> System.AggregateException: One or more errors occurred. (Authentication to host '127.0.0.1' failed. (I/O error occurred.))
 ---> System.AggregateException: Authentication to host '127.0.0.1' failed. (I/O error occurred.)
 ---> System.IO.IOException: I/O error occurred.
   at MySql.Data.Common.Ssl.<StartSSLAsync>b__11_0()
   ...
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

dotnet run ended with exit code 134, about 15 seconds after the burst began.

What I Checked

The password and the server. Both were fine: 110 requests at a time worked on the same settings, and so did a request after the crash, once I restarted the API.

Whether it repeats. Not every time, which makes it easy to miss in a test. I ran this burst 26 times on a freshly started API. The process ended in 17 of them. In 4 more it stayed up, but gave dozens of 500s and 503s, over 20 to 150 seconds. Only 5 got through cleanly. A pool that had already grown to 50 connections survived it once too. So it depends on timing.

The stack trace. It starts at TimerQueueTimer.Fire, on a thread pool thread. A timer, not a request, threw this exception. That's why my try/catch and my exception handler never saw it.

The driver's source, tag 26.10.0. Three lines explain it. When the pool is full, a request blocks its thread, inside an async method:

MySqlPool.cs

// We have no tickets right now, lets wait for one.
if (!_autoEvent.WaitOne(timeOut, false)) break;

Opening a connection starts a timer of Connect Timeout seconds:

Driver.cs

if (settings.ConnectionTimeout != 0)
  connTimeoutSource = new CancellationTokenSource((int)settings.ConnectionTimeout * 1000);

And during the TLS handshake, the cancellation callback throws:

Ssl.cs

using (cancellationToken.Register(() => throw new AggregateException($"Authentication to host '{_settings.Server}' failed.", new IOException())))
  await sslStream.AuthenticateAsClientAsync(_settings.Server, certs, tlsProtocol, false).ConfigureAwait(false);

The threads. I counted them with ps -M during the burst:

Threads in the API process

Before the burst   31 threads
After 2 seconds    39 threads
After 4 seconds    43 threads
After 6 seconds    46 threads
After 8 seconds    50 threads
After 10 seconds   52 threads
After 12 seconds   57 threads
After 14 seconds   the process is gone

That's about 2 new threads a second. Microsoft's thread pool starvation guide describes this pattern: the pool "has no available threads to process new work items".

Whether threads really are the bottleneck. I allowed the pool 500 threads from the start, and the same burst finished in about 2.4 seconds with every request answering 200, in all 8 runs. So yes.

Why It Happens

Each piece is harmless alone. Together:

1. More requests arrive than Maximum Pool Size. The extra ones block thread pool threads while they wait.
2. .NET adds threads slowly, so new work queues up. That includes the handshakes of connections the pool is still opening.
3. Connect Timeout runs out while a handshake is still waiting for a thread.
4. The cancellation callback throws on the timer's thread. Nothing is above it to catch the exception, and .NET ends the process when that happens.

The Fix

Never let more requests reach the pool than it holds. A request uses one connection at a time, so a concurrency limit equal to Maximum Pool Size does it. The extra requests wait in the limiter's queue, and that wait doesn't hold a thread.

Program.cs

using System.Threading.RateLimiting;
using MySql.Data.MySqlClient;

var maximumPoolSize = (int)new MySqlConnectionStringBuilder(
    builder.Configuration.GetConnectionString("Inventory")).MaximumPoolSize;

builder.Services.AddRateLimiter(options =>
{
    options.RejectionStatusCode = StatusCodes.Status503ServiceUnavailable;

    options.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(_ =>
        RateLimitPartition.GetConcurrencyLimiter("database-connections", _ => new ConcurrencyLimiterOptions
        {
            PermitLimit = maximumPoolSize,
            QueueLimit = 500,
            QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
        }));
});

// After var app = builder.Build(), and after your error handlers:
app.UseRateLimiter();

Past 500 waiting requests, the API answers 503 straight away. How to Limit Concurrent Requests in ASP.NET Core explains each setting.

Raising the thread pool's minimum also stops the crash, as How to Raise the Minimum Threads of the .NET Thread Pool shows. It costs far more threads, though, and it doesn't limit the wait.

After the Fix

The same burst:

Output

 600 200

I ran that 8 times, and all 8 answered every request, in about 2.4 seconds. With a pool of only 5 and 500 requests at once, all 2,000 requests answered 200 too. And with a pool of 25 and 500 at once, the process finished with about 39 threads, where the raised minimum ended with about 400.

Common Mistakes

Mistake	What happens	Do this instead
Adding try/catch around the query	The exception is thrown on a timer's thread, not yours	Stop the cause
Raising Maximum Pool Size	The crash moves to a higher number of requests	Limit requests to the size of the pool
Setting Connect Timeout=0 to remove the timer	Every pooled connection then fails	Keep a number
A limit higher than the pool	Requests still wait inside the driver	Read the limit from the connection string

Troubleshooting

Symptom	Likely cause	Fix
It still crashes, with the limit in place and no burst	A handshake stalled on the network	See the FAQ
503 straight away under load	The queue is full	Raise QueueLimit, or find what's slow
Timeout expired … prior to obtaining a connection from the pool	Requests wait inside the driver	The fix

FAQ

Can it happen without a burst?

Yes. I wrote a small proxy that passes MySQL's greeting and then goes silent, like a network that stalls during the handshake. The process ended exactly when Connect Timeout ran out, even with Connect Timeout=3 in a small console app, and a health check's timeout does the same. The window is small, because a handshake takes milliseconds. Nothing in the app can catch it, so run the API under something that restarts it when it stops.

Does a client disconnecting cause it?

No. That cancels the same token, but on Kestrel's thread, and Kestrel catches the exception and logs it. I saw fail: Microsoft.AspNetCore.Server.Kestrel[13] in the log, and the API kept running.

Does AddDbContextPool help?

No. It pools DbContext objects, not connections. How to Use AddDbContextPool with MySQL in EF Core has the measurements.

Conclusion

Authentication to host failed. (I/O error occurred.) from a timer's thread isn't about your password. It's what happens when Connect Timeout runs out during a handshake, and a burst above Maximum Pool Size starves handshakes of threads. Put a concurrency limit in front of the pool, equal to its size, and keep something around that restarts the process.

Additional Resources

• Debug ThreadPool starvation — what blocked threads do to an app.
• Rate limiting middleware in ASP.NET Core — the concurrency limiter and its queue.
• MySQL Connector/NET source, tag 26.10.0 — MySqlPool.cs, Driver.cs and Ssl.cs.

Related

• How to Limit Concurrent Requests in ASP.NET Core
• How to Size the MySql.Data Connection Pool
• How to See Thread Pool Starvation with dotnet-counters