I meant to write this almost a year ago, but then I was asked to take a look at high cluster service privileged time usage and realized I had forgotten! I guess better late than never?
If you’re using WSFC integrated AGs (cluster_type = WSFC) then the cluster database is the location in which availability group data is stored, which is essentially, the registry. When you run queries against the various DMVs and catalog views associated with the current state of the AG, a database in the AG, backup preferences, etc., almost all of them will need to reach out to the cluster to get the requested data – simply because the information is not stored in SQL Server. This is generally not an issue, for example you’re doing a log backup, checking if you’re the primary replica to know if you should do the backup or not. This shouldn’t be executing 100 times in parallel every few seconds, maybe once every 5 minutes is more the average.
This can, however, become a problem if the data is asked for in a highly parallel operation – such as having 300 databases in one or more availability groups and all log backups kicking off, in parallel, at the same time, where they each will do a check to see if they are the primary replica and should continue. This will lead to many calls to the cluster service to read the data from the cluster database and let SQL Server know the values so that SQL Server can go do whatever it is it was asked to do, which unfortunately can get all blocked up in concurrency and eats a ton of privileged time while doing it. The registry was never supposed to be some high-speed data store, but this is merely an implementation detail and one that I wish people didn’t have to specifically understand… and most don’t, because most of the time there isn’t a problem. Most people don’t believe that slamming their system with a large amount of concurrent heavy hitting requests is a good idea, some do though and that’s why I have this blog post, so there’s that.
I am picking on backups because they are the easiest thing to not think about, even log shipping by default will run a check before taking a backup, because that’s the correct thing to do. I’ve watched other things do this, though, and the root cause for the most hilarious of these was the _entire_ DBA team had the AG Dashboard in SSMS open and refreshing every 5 seconds. So yeah, it’s nothing I would expect people to think about until they hit an issue (which, isn’t that most things?) and then it’s going to look as though the cluster is at fault when it’s merely the messenger in this situation.
The easiest way to see if this is the cause is to look at registry usage via something such as procmon on the AG resources, if they are being hit in quite large numbers, here’s your culprit.