Jump to content
BGags

How I got to 95% patch efficacy in Eight Easy(?) Steps

Recommended Posts

EDIT: There's a newer article over here that's less out of date but doesn't have all the scripts or thread content yet: http://www.labtechgeek.com/forum/viewtopic.php?f=7&t=3559

 

LabTech is pretty good at delivering Windows Updates right out of the box, but it's far from perfect. Some of its behaviors are even questionable, such as the "Pushed" flag on a patch (more on that later). Even though LabTech 11 promises to fully re-vamp patch delivery, there's still a lot to know from a purely OS standpoint on how Windows Updates are actually delivered. So, here's what I've learned, and the practices I employ to keep the unofficial title of Patching God:

 

1. Prune your agents.

 

This is something that we could all probably do better. If an agent has been offline for more than 30 days, find out why. Did the agent fail? Discover if it's still alive by ping, tracert, ARP cache, AD query, any way you can. Is it dead? Yeah? Delete it! Not only is this good for your patch health reports, it'll take a load off your LabTech server if it doesn't have to run all those offline health checks on dead agents.

 

2. Patching once a week simply isn't enough.

 

We have auto-join groups called Patch Remediation, where I fire off scripts that install updates with no restart 2-3 times a day. I have yet to hear a user complaint about this as long as Windows Update is running okay (more on that later).

 

The scripts run against auto-join groups whose search criteria include agents that are a) missing approved updates, b) don't have any of the Ignite "disable patching" flags set, and c) have permission to do Patch Remediation. By c), I mean I have an EDF for the clients that only want me to deliver patches after hours, so I use a checkbox for those weirdos. I mean, who doesn't want to be updated and secure? Silly.

 

I have separate remediation schedules for servers (which don't need it as much, since they're on all the time), workstations (early in the morning and just before end of day), and laptops (special windows at lunch, and again at 8pm where people are working from home after dinner). I cannot overstate how simply effective this has been.

 

3. The Windows Update Agent is a really big deal.

 

I cannot stress this one enough.

 

For example: Server 2008 R2 SP1 and Win7 SP1 both emerge with a default Windows Update Agent of 7.6.7600.256. If you rely on the LabTech command of "Install the Windows Update Agent", this is as high as you go, and if you check for available Windows Updates with this WUA version you'll see about 50 updates. To see the other 200+ available, however, you have to be using at least 7.6.7600.320, which came out in November 2014, but suffers from high CPU utilization during hotfix inventories.

 

Told you it was a big deal.

 

The second-latest WUA (September 2015) for Win7 and Server 2008 R2 is version 7.6.7601.18979, and it does okay. The inventories take a long time, but they don't suck down memory or CPU that much. This WUA update is available via KB3083324. There's an even newer one, released October 15, 2015, that I haven't finished testing yet (in the words of Han Solo "I just got this bucket back together, I'm not gonna let something tear it apart").

 

You might be wondering how I get the WUA version, and what I do about it. If you guessed "EDFs and Scripting" to be a part of the solution, you're right. But first, the goods on the WUA version:

 

POWERSHELL: (Get-Item "%windir%\system32\wuaueng.dll").VersionInfo.ProductVersion

 

That's it. You're getting the Product Version attribute of the WUA engine DLL file and returning it through PowerShell in the script engine. You can then populate an EDF and check against it by maintaining a constant somewhere.

 

I then took that and wrote a remediation script that matches OS with target WUA versions and installs what's missing. The script has global variables for OS WUA versions and OS WUA download URLs. This is kinda painful to maintain, but whatever. Note that some WUA updates require a restart, so you'll want to factor this into your scheduling. I use searches and auto-join groups for a self-maintaining process, and run the actual WUA remediation script twice a week during business hours on needed systems. Nobody has yet to notice.

 

4. Windows Update breaks fairly often. Repair it automatically.

 

(see below note on scheduling if you see 0x8024001E show up too often)

 

I'm not actually going to post my Windows Update Repair script since it references a lot of EDFs that won't exist for you, but here's the basic run-down of what it does:

 

- Check the Windows Update Agent version (see above) and update if needed (and bail if reboot needed after update).

- Check for and remove any WSUS policies applied (various registry entries) *

- Use stock LT commands to Set WUA To Defaults, then Set to LabTech Mode

- Nuke the %windir%\SoftwareDistribution\*.* contents (after stopping wuauserv) **

- Nuke the %windir%\system32\catroot2 folder (after stopping cryptsvc)

- Reset the winhttp proxy (Hey, it can't hurt, right?)

 

* You know Those Clients with that SBS 2011 server that was set up with all the defaults and WSUS policies that they never bothered to actually keep up with? I have an EDF and a Get Info script pair that looks specifically for agents with WSUS policies set up so we can go in and uninstall WSUS. LabTech patch approval is so much better.

 

** The biggest complaint you'll get from this is that it nukes the Windows Update history people see in the WU control panel. Sometimes you'll get a client who looks at the Windows Update control panel to see if you're doing your job, and they'll see that no updates have been installed, or the list of installed updates is very limited. Then they call you with indignation and vitriol. Just tell them to go through Programs and Features, then View Installed Updates, to see that everything's installed. Then tell them that your LabTech product does this because it's more effective than Windows Update, and they'll eat that up.

 

Anyway, that's the easy part. The tough part was figuring what criteria to use to determine if something's broken. After doing a bunch of crunching, I decided on flagging the need for repair after either of two major criteria on one agent:

- Three failed hotfix inventories in a row no subsequent success

- Two failed attempted updates in a row no subsequent success

 

The details on how I fashioned EDFs, scripts, queries, groups, and monitors got ugly pretty quickly. To strip it down: I've got one script that runs in the evening against all agents under patch contract that checks these criteria and sets or clears my "Update Repair Needed" EDF. Then the repair script above runs during the day against any agent with that flag set. If the repair script got all the way through the repair (no reboots needed, all commands completed successfully), the flag is cleared and an Informational alert is generated that the repair has been run.

 

From there, you fire up a different monitor that looks for systems a) with missing, approved patches, and b) on which Windows Update Repair has already been run (remember the alert?). That's when you create your ticket. 'Cause if you're still having problems after all that fixin', that's when you need... a technician'? I can rhyme, some of the time.

 

5. Get rid of the "Pushed" hotfix status

 

The "Pushed" status for a patch means that LabTech has tried twice to install the patch and it didn't succeed, so it's not going to try to install it anymore. That's literally what it means, straight from the developer's mouth.

 

Are you kidding? LabTech! Don't give up! You can do it!

 

That's the behavior. And for my environment, it doesn't work. It's not effective, it gets in the way, and it lowers my patch efficacy numbers. You can't actually "get rid of" this functionality, but you can certainly reset the Pushed flags. I do it directly with a SQL query every night with a stored procedure. Here's the SQL:

 

UPDATE hotfix SET pushed='0' WHERE installed=0 AND pushed=1 AND approved=1

 

Before you do this, if you want to see how many patches you're missing due to the Pushed flag, here's the SQL to get that count:

 

SELECT COUNT(hotfix.`HotFixID`) FROM hotfix WHERE installed=0 AND pushed=1 AND approved=1

 

Yeah. You thought you slept well at night.

 

6. Make sure you get your schedules straight.

 

The most common error code I've seen when patches or hotfix inventory fails is this one: 0x8024001E ...which is WUA code for "I'm busy doing something else, try again later." Are you patching when you're also doing a hotfix inventory? That's gonna cause problems. Are you doing a hotfix inventory right at 5pm when all your workstation agents are shutting down? Ha! Try again! This might take some coordinating and planning, but basically you want to make sure you're only doing one thing involving Windows Updates to any given Agent at any one time.

 

7. Don't forget non-OS Microsoft products.

 

You know when you go into the Control Panel and you get that bit on the screen that says "You receive updates: ...For Windows and other products from Microsoft Update"? Did you know you can set that programmatically?

 

It's ugly, but here's a line of PowerShell that you can throw into a script to do this for any supported Microsoft OS:

 

$ServiceManager = New-Object -ComObject "Microsoft.Update.ServiceManager"; $ServiceManager.ClientApplicationID = "My App"; $ServiceManager.AddService2( "7971f918-a847-4430-9279-4a52d1efe18d",7,"")

 

Hey, don't blame me, I didn't write the OS.

 

8. Ticket against patch efficacy, not against encountered errors.

 

After all the automatic pushes, repairs, alerts, flags, and all the other crap I do, in the final analysis I found it most effective to only create an actual ticket for action based on the following criteria:

 

a) A system is missing 5+ approved patches

b) That system has tried to apply patches within the past 30 days (success OR fail)

c) Windows Update Repair has been run successfully (we look for that alert in the past 30 days)

 

The tickets get generated by a Monitor that runs on the first of the month. Why the first? Because it's the week before Patch Tuesday, that's why. By the beginning of the month, all our patches should have had a chance to make their way onto all our computers. If the above three criteria have been met for one agent, chances are you've found a system that really requires some serious Windows Update surgery.

 

So, hear me out: You have to be able to trap for systems that are failing to apply updates in ways that LabTech can't easily spot. Because really, when Windows "installs" an update that needs a reboot, it's not really running the full install; it's staging the update for installation during system restart. If staging is successful, LabTech will report a successful update install. But if that update fails and the system has to revert, the Agent won't see the failure directly because it's not running. Although this failure leaves footprints in the Event Log, I've not found those to be reliable. Maybe I'll create a monitor for this particular problem one day, but I've been doing fine without it.

 

And really, you ultimately don't just want to track errors, you want to be tracking patch efficacy. You want to have tickets at the beginning of the month that let your client know that YOU know that one of their laptops is missing seven updates and our automated system couldn't fix it because the laptop keeps going to sleep and would you please turn it on for 45 minutes so I can slam it with scripts?

 

The proof:

 

When it comes to numbers, my servers are 98.7% patched, my workstations 96.6%, and my laptops 89.9% (DAMMIT) for all agents that have checked in within the last 30 days. I think that's pretty good.

 

Another bit of information: All Microsoft support calls relating to Windows Update delivery are free. One hundred percent free. No kidding.

 

Hey, want a script?

 

Here are two of the scripts I described above. The "WindowsUpdateRepairCheck" is described in section 4 above, sanitized without EDFs (using Script Note in its place). This is the script that I run against every agent in the "Under Microsoft Patch Contract" group every night that sets a flag signaling the agent is in need of Windows Update repair. It's an offline script that does its logic just against the database. Feel free to screw around with it.

 

The "WindowsUpdateRepair" script is the script that goes around fixing stuff. It's also been sanitized without subordinate scripts or EDFs. Be sure to check out the Globals tab to see what's defined as my preferred Windows Update Agent version for Win7 / Server 2008 R2. Also note that it creates an alert on your agent after it's run successfully; if you don't want that, disable or delete that function.

 

May the Patch be with you.

WindowsUpdateRepairCheck.zip

WindowsUpdateRepair.zip

Edited by Guest
  • Like 1

Share this post


Link to post
Share on other sites

Rather than SQL, which I'm not too familiar with, I followed the advice in this KB article to use a dataview for the same purpose of finding the "Pushed" patches.

Edit: Including the KB article I refer might be handy. https://docs.labtechsoftware.com/knowledgebase/article/7285

I had 6,287 of them.

 

Thought I slept well, indeed. Thanks for your this massive post of brilliant insight!

 

I don't know if much will come of it on our end, but I'll definitely put this into consideration. In addition, if you would be willing to post the results of your testing (if/when you do it) with the October 2015 WUA, I'd be grateful. I'm hoping that "update to the latest WUA" will be the easy fix to the high CPU use issue we are seeing.

Share this post


Link to post
Share on other sites

I just posted in a different thread about the High CPU use problem as well! Let me... here, have a link:

 

viewtopic.php?f=7&t=1880&start=25#p13149

 

And here's the relevant text from the post:

 

"You know what I found was doing it? A Windows Update Agent version of 7.6.7600.320, plus the LabTech Windows Update setting of "Remove user access to Windows Update".

 

I'm not even kidding.

 

Do the following:

 

1. Set Windows Update to Defaults

2. Set Windows Update to LabTech mode

3. Install KB3083324 on every Windows 7 and Server 2008 R2 agent you have. This will require a reboot."

Share this post


Link to post
Share on other sites

Nice!

 

Someone should take all this knowledge and package a plugin around it to do patch remediation. I could see a nice main interface that let you adjust the different timings and set different levels of remediation and some data views to represent the different statuses of patching.

 

Cubert :ugeek:

Share this post


Link to post
Share on other sites
Nice!

 

Someone should take all this knowledge and package a plugin around it to do patch remediation. I could see a nice main interface that let you adjust the different timings and set different levels of remediation and some data views to represent the different statuses of patching.

 

Cubert :ugeek:

 

With the speed you turn out plugins Cubert, I guess we will see one in a day or two... :lol:

Share this post


Link to post
Share on other sites

A gift from me to you!

 

I've commented the heck out of the script that I run nightly against all patching agents to check if they need a Windows Update repair. The idea for this is referenced in the second half of Item 4. above, when I talk about the criteria for issuing the repair script. I've replaced the three calls to Set EDF for "Windows Update Repair Needed" with comments in the appropriate places. It's an offline script that doesn't actually touch the agent, so feel free to play with it.

 

[EDIT] I just threw this into the body of the main post as well.

WindowsUpdateRepairCheck.zip

Edited by Guest

Share this post


Link to post
Share on other sites

Here is a quick preview of what I got so far,

 

Master switch turns on scanner which every four hours will probe Windows systems for WUA versions and report back. We then graph the different versions found per OS type. If everything is perfect this should always be 1 per OS, if you have more than 1 then you have different versions deployed across environment. Next we show you how many "Critical" patches have failed to install and how many are 3 or more critical missing. Then show you how many patches today were installed.

 

We are going to add a tab or 2 of different graphs to give users a deep view of current patch status across clients.

 

I am also working on a mediation peice that will update all WUAs to bring all systems in to compliance. I will also add a counter for Pushes and a methed to clear the pushes whit a quick click. Then finally like to add in a WUA repair tool that helps resolve WUA issues

 

 

Cubert :ugeek:

Capture.PNG.618c41831d6ae6a78a65e830096f8e29.PNG

Share this post


Link to post
Share on other sites
Here is a quick preview of what I got so far,

 

Master switch turns on scanner which every four hours will probe Windows systems for WUA versions and report back. We then graph the different versions found per OS type. If everything is perfect this should always be 1 per OS, if you have more than 1 then you have different versions deployed across environment. Next we show you how many "Critical" patches have failed to install and how many are 3 or more critical missing. Then show you how many patches today were installed.

 

We are going to add a tab or 2 of different graphs to give users a deep view of current patch status across clients.

 

I am also working on a mediation peice that will update all WUAs to bring all systems in to compliance. I will also add a counter for Pushes and a methed to clear the pushes whit a quick click. Then finally like to add in a WUA repair tool that helps resolve WUA issues

[attachment=0]Capture.PNG[/attachment]

 

Cubert :ugeek:

 

You never stop to amaze.

 

Thank BGags for this. Very insightful.

Share this post


Link to post
Share on other sites

For what it's worth, I threw the October version of WUA on a test Server 2008 R2 VM, and the processor use looks a little bit dicey to me.

 

I don't have a screenshot of the September WUA on the same machine, but it spiked the CPU a lot less.

 

Here are the details on the October WUA:

WUA Version 7.6.7601.19016 - Oct 2015 - KB3083710

https://support.microsoft.com/en-us/kb/3083710

59ec943a66ed6_2015-10-2917_20_11.jpg.76ac8b3a12f457efa5193a2f9b3da88b.jpg

Share this post


Link to post
Share on other sites
Here is a quick preview of what I got so far,

 

Master switch turns on scanner which every four hours will probe Windows systems for WUA versions and report back. We then graph the different versions found per OS type. If everything is perfect this should always be 1 per OS, if you have more than 1 then you have different versions deployed across environment. Next we show you how many "Critical" patches have failed to install and how many are 3 or more critical missing. Then show you how many patches today were installed.

 

We are going to add a tab or 2 of different graphs to give users a deep view of current patch status across clients.

 

I am also working on a mediation peice that will update all WUAs to bring all systems in to compliance. I will also add a counter for Pushes and a methed to clear the pushes whit a quick click. Then finally like to add in a WUA repair tool that helps resolve WUA issues

[attachment=0]Capture.PNG[/attachment]

 

Cubert :ugeek:

 

Cubert you are a genius! What would the LT community do without you!

Share this post


Link to post
Share on other sites
A gift from me to you!

 

I've commented the heck out of the script that I run nightly against all patching agents to check if they need a Windows Update repair. The idea for this is referenced in the second half of Item 4. above, when I talk about the criteria for issuing the repair script. I've replaced the three calls to Set EDF for "Windows Update Repair Needed" with comments in the appropriate places. It's an offline script that doesn't actually touch the agent, so feel free to play with it.

 

[EDIT] I just threw this into the body of the main post as well.

 

 

BGags, I can't tell you how much this is appreciated. We have been struggling with getting our patching up to date for months.... I've been accessing systems until 1 and 2am and manually running Windows Update..... getting REAL tired of that!

Share this post


Link to post
Share on other sites

This is AWESOME !! Good info ! We haven't started patching through LT yet. This will definitely help us to be efficient.

Share this post


Link to post
Share on other sites

Thanks for the informative post! Spurred me to dig into this and try to emulate a similar process.

 

Question on the WUA version. Does it only matter for windows 7 and 2008 server? I checked the version # on a 2012 box and it was a much higher revision. Do you have a separate method for dealing with newer OS, or is WUA simply more reliable on them?

Share this post


Link to post
Share on other sites

Well here is a demo of what I have so far.

 

I neutered the Update so do not expect that to work in this release. I am looking at ways to make that better. I have included the autoplugin code so as I update the plugin you will get the new versions automatically.

 

http://lp.squidworks.net/index.php?PRODUCT=887F1867-86E4-48B9-95DC-EE79AC12E6F0

 

 

Have fun...

 

Remember to restart the LT DBagent and your console after adding DLL.

 

 

Cubert :ugeek:

Share this post


Link to post
Share on other sites
Thanks for the informative post! Spurred me to dig into this and try to emulate a similar process.

 

Question on the WUA version. Does it only matter for windows 7 and 2008 server? I checked the version # on a 2012 box and it was a much higher revision. Do you have a separate method for dealing with newer OS, or is WUA simply more reliable on them?

 

Matt (may I call you Matt?),

 

I've found that the WUA available for the Windows 8 / Server 2012 class of operating systems is simply more reliable and doesn't impact the CPU as much. There have been updates to the WUA for these operating systems, but they've typically emerged as Critical Updates with no prerequisites, so I just approve them for installation and they make their way out to the agent without any extra effort. Since it's no extra effort to collect the WUA version for these servers, the collection and health scripts still run against everything.

 

Keep in mind that Server 2012 and Windows 8 are somewhat "leaner" operating systems in terms of their footprint, and there are far fewer updates available for those systems, so there are simply fewer data points to search when doing a hotfix inventory. With 8.1 / Server 2012 R2, the Windows Update Agent is more advanced still. In fact, the WUA for 8.1 is one of the reasons that the LabTech 2013.1 minor update even made an appearance.

Share this post


Link to post
Share on other sites
Well here is a demo of what I have so far.

 

I neutered the Update so do not expect that to work in this release. I am looking at ways to make that better. I have included the autoplugin code so as I update the plugin you will get the new versions automatically.

 

http://lp.squidworks.net/index.php?PRODUCT=887F1867-86E4-48B9-95DC-EE79AC12E6F0

 

 

Have fun...

 

Remember to restart the LT DBagent and your console after adding DLL.

 

 

Cubert :ugeek:

 

CUBERT! YOU'RE A MONSTER!!!! This is tremendous. What a thing! I've gotta give it a try, and probably will over the weekend. I might make suggestions, but probably not many of them.

 

Way to go, man.

Share this post


Link to post
Share on other sites

That makes sense for the WUA. Always found it funny when you had to update windows update.

 

Slowly emulating your steps; even if Cubert's plugin replicates it this is good practice with scripts/groups/searches.

 

Right now I have a search that looks for Approved v_hotfixes = 1 and Installed = 0 to see what machines have patches approved but not installed. I've also made a script based off your PS code to check the WUA version and write it into an extra data field, which i'm referencing in another search to display any computers with the version # less than the September patch.

 

I think the next step is to set up a script to update the machines with out of date WUA. Then presumably I can see my true missing patch counts.

Share this post


Link to post
Share on other sites
I'm not actually going to post my Windows Update Repair script since it references a lot of EDFs that won't exist for you, but...

 

Would I be an awful person if I asked you to share it anyway? It seems to me that building off of your work - even if I have to rearrange EDFs - is easier then reinventing the wheel here.

(I confess, I'm lazy, but isn't that the point of automation?)

 

If I make any improvements I'd be more than happy to share them with the collective in turn.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×