Possible configuration error: 1000000 IO requests allocated

You’ve won!

If, somehow, you’ve managed to see this error in your errorlog then congratulations, you’ve won an instance of SQL Server that probably won’t be doing much.

I found out about this message a few months ago, but it has been in the product for years and I went this long without ever even knowing it existed (congrats me!) until I was asked about it and coincidentally ended up finding it in an errorlog the same week. Clearly, I have too much fun packed into my weeks. I asked around, only one other person had ever found this in an errorlog before… that’s either impressive, depressing, or some perfect quantity of both – mellow it out to a smooth melancholy.

Skipping The Boring Details

The message is mostly self-explanatory in what it attempts to convey – which is there are a whole big bunch of IO requests that were given out, and there may possibly exist some sort of configuration error. In so far as errorlog messages go, this isn’t the worst, but not even close to the best, it really could be done better. Regardless, let’s break this one down, though it should never occur in any environment. If it does, there is definitely a problem, who knows if it’s a configuration issue though… more on that.

This breaks down to two things, outstanding IO requests and magic numbers. SQL Server, like any other application that is written to use asynchronous IO on Windows [or emulations of a Windows environment running in a self-contained process with an abstraction layer on top of it (that’s a SQL on Linux joke, everyone)], SQL Server creates overlapped structures, fills in the request, and then asks Windows to do the IO (contrary to some popular belief that SQL Server has its own IO stack that doesn’t go through Windows, I’m not joking this time). SQL Server tries to be efficient at what it does, especially given that SQL likes to ask for a bunch of data, regardless of where it lives. Since this means it’s going to potentially create and fill out many of these structures along with SQL Server specific information that may need added or tagged onto the request, it would make more sense to just reuse the structures after they complete. This helps efficiency in many places and hopefully saves some processor cycles and memory usage. In comes the IORequestDispenser.

The job of the IORequestDispenser is exactly what it sounds like, to dispense IO request related items and structures. To be efficient, once the data structures are created it makes sense to reuse them, so as new ones are needed, they are created but if an IO request has been satisfied and completed then the structure can be re-used without having to create a new one. This means there can be a small cache of requests that have been completed that can be reused without having to create new ones, which means even under load and stress there should be an upper limit to how many requests are created as the cycle of completing requests can be reused to satisfy new requests. The error states that this has gone above 1,000,000 (that’s 1 million) requests that were created. That’s a lot.

Why 1 million? No idea. Someone probably picked a large number and thought, “wow, if anyone goes over this number it must be a big issue” and so used said number.

Long Story Longer

This all wraps up to basically say, if you see this message in your errorlog (which will only be printed once for the life of the process) it may or may not actually be some sort of configuration error. Chances are it probably isn’t. The one time I did see this in an errorlog was when a very large server (416 cores) lost connections to the storage for a period of time (a few minutes) and had a *massive* workload which requested – you guessed it – quite a lot of reads and writes to files. Based on my large sample size of 1, I’d say this is more likely to be an issue with storage availability and or performance than a configuration issue.

For those of you playing along at home, this particular scenario ended up dumping on a latch timeout which is not at all a surprise when your storage isn’t completing the requests given in a timely manner.