Issue Introduction
Someone asked if I’ve ever had a CU install where you run it and it goes through the extraction process, then right as it hits 100% just exits and nothing happens. Well, that’s pretty weird, and no, I hadn’t. I was, however, intrigued! Since I love my readers, I made a repro of what the person this was occurring to, saw.
Questions That Need Answered
- Is this an issue with the CU? After all, this wouldn’t be the first time.
- Why is this happening?
- Is there a fix so it can work?
Initial Data Collection
We already know a few things but first let’s go over how the overall CU process works. CUs, when downloaded, are just self-extracting installers. This means the SQL Setup process and the needed files for it are just compressed and signed for this CU package. When you run the CU installer it unzips and extracts the files to a temporary location. Once extracted the setup process should be launched from the temporary location. So now that we know the overall process, we can conclude that the person running the CU has permissions enough to execute it (admin) and that at least the extraction part seems to more or less work. This means we are having problems somewhere between the extraction of the files and launching of the setup executable. Given this, a good place to start would be to gather a procmon of the entire process, from extraction start to finish. The person collected a procmon of the whole process, from start until counting to 10 after the extractor vanishes. It’s always good to wait a few seconds to minute or so when these types of issue occur as there can be ancillary processes or items that kick in a few seconds after an issue occurs that helps give telltale signs.
I was also told that Crowdstrike, among other items, was installed on the server. I’m a pretty big hater of that particular piece of software, and while I’d love to blame it without doing any work on the problem, I don’t think that’s fair. Not everything is a Crowdstrike issue, just 99% of things.
Checking The Collected Data
Now we have the procmon data… but what do we look for?
I have a few standard things I like to complete when handed a procmon, even if the symptoms don’t make it seem needed. The first is I like to aggregate the operation results and get a count of them, it’s gives for a nice overview of any potential issues, such as access denied. Next I like to check out the cross reference summary. In this case, neither were helpful in identifying any specific issues.
The next logical step is to find the end of the extraction process and see if we’re able to even launch the setup process. Failing to be able to launch the setup process, say because of an anti* software installed would be a pretty damning piece of information. You can filter only on process and thread activity, which makes this effort much easier than it used to be, and we can quickly see that not only does the setup launch, but it runs for a short period of time.
Ok, so setup _does_ launch, that’s odd we don’t see the splash page or window for it. So let’s go see where it exits, because if it’s waiting for a resource or some other item is should just kind of sit there versus exiting. Finding the associated process exit we can see it doesn’t run for long and we get an exit status of -2067529723! Finally some good information. If you see a status or code with a negative starting value, it’s because it’s actually unsigned and either an NTSTATUS or an HRESULT, so converting this to hex gives the typically used value, in this case it’s 0x84C40005 which isn’t as helpful as initially thought.
Right. Time to keep thinking. We know the files extract, we know that setup runs but exits. Typically the first thing the setup process does is start to write a log file to the Temp directory so let’s go look to see what happened with that, if it’s there we should get a copy of it as it’d have some great information in it. Looking at the procmon again we get a major clue, the log file was never created and the result was PATH NOT FOUND!
If setup fails to create the log file, it will indeed exit. This is all making sense now, immediately we need to check the environment variables for the process create, which is captured in procmon! We can see, in fact, TEMP and TMP are horribly incorrect.
Mitigations and Results
Truly the mitigation and fix is setting the TEMP and TMP variables to point to _real_ locations that are used for holding temporary items. After this was completed, the CU proceeded normally and installed without issue.
Bringing It All Together and Answering Questions
- Is this an issue with the CU? Afterall, this wouldn’t be the first time.
A: No, this was a configuration issue where the environment variables were improperly set. - Why is this happening?
A: Because the environment variables aren’t properly set. Why? Well, in this case, there was a process that ran on the system which the person did know about. This process overwrites various environment variables and these were two that were overwritten. - Is there a fix so it can work?
A: Yes, making sure your environment variables are correct. This is purely a local configuration (and in my opinion, process) issue!
Closing Thoughts
I really didn’t see this ending up to be an environment variable issue. My first thought after hearing Crowdstrike was that it was doing something very bad – or just business as usual for it I guess – and not letting the process spawn, or injecting itself into the process and causing other issues. I’m pleasantly surprised in this case that it had nothing to do with that!
This is but one of many cautionary tales of not having your configuration or environment correct and ending up with some weird issues. It’s not the first time, it won’t be the last, but what really makes me do a double take on this is that the person knew (and failed to tell anyone) that there were processes which were overwriting existing environment variables. That seems like something you’d tell someone, especially when everyone’s favorite phrase is, “we didn’t do anything, nothing has changed.” Yeah, nothing changed except all these things which we’ll fail to mention. Cool. Fun. /Heavy Sigh