How to Properly Update Your Remote Desktop Services Environments

Windows OS and other application updates have the potential to wreck your RDS deployments, as I discussed at length recently at my RDPHard Twitter feed. Last year, Microsoft’s QA team let through several buggy OS updates that caused high impact problems for RDS environments. That said, you can update Remote Desktop Services with less drama and more confidence when you test those updates in an RDS staging deployment first. Please review this article and watch my comprehensive RDPHard Channel video below to understand how setting up an RDS staging deployment will make your life easier.

Step 1 – Create an RDS Staging Deployment That Receives Updates First

Here are my recommendations for how to set up an RDS staging deployment to smoothly update Remote Desktop environments.

• Create it in the same Windows domain as your production RDS deployment, but with your staging RDS VMs located in a separate OU (so you can change things like GPOs, FSLogix configurations, etc without affecting your primary RDS deployment). If you’re an MSP, you can just create a separate test tenant in its own domain with its own RDS deployment.
• The RDS staging deployment should contain each role service you’re running in your production deployment. So if you have gateways in production, install at least one gateway in the staging deployment. The same thing goes for RDWeb, connection brokers, etc. One exception, however, is that you can use the same RDS licensing server(s) as you’re using in production to service your staging deployment.
• In the staging deployment, you should provision at least one session host for RemoteApps, and one session host for Desktops.

Step 2 – Apply Updates to the RDS Staging Deployment First, Using Technologies Like WSUS or Similar

Once you have an RDS staging deployment built out, you can use technologies like Windows Server Update Services (WSUS) to push Windows and application updates out to your guinea pig servers first. If anything breaks, you’ll see it first on the staging servers, and you can delay the deployment of offending updates until you have a mitigation plan in place. This is much easier than having to roll back already applied updates in production. And in many cases, like that Server 2022 Update from last Spring that wiped the connection broker role completely off of servers altogether, just rolling back an update might not undo the damage that was caused.

Step 3 – Use Manual Testing, Automated Testing, or Both to Test the RDS Staging Deployment Immediately After Updates Are Applied

So, once you have your RDS staging deployment built, and once you have a way to apply updates to that guinea pig deployment first using WSUS or equivalent, how do you test applied updates? Well, you can do manual testing, automated testing, or a combination of both.

With manual testing, you or other users can log in through the gateway and broker of the staging deployment and start running applications on your assigned session hosts, to see if anything goes wrong. In fact, you can incentivize some of your power users, who are capable of reporting errors to you, to use the staging deployment as their primary RDS farm, by wooing them with promises of higher performance, since that staging farm will have fewer users on it than your production environment. If they encounter breaking changes, they can report them and then log into the production collection instead, provided you’re using a user profile solution like FSLogix which can span multiple collections. That stands in stark contrast to user profile disks (UPDs), which are tied to specific collections.

With automated testing, you can use a product like our Remote Desktop Canary, which does synthetic login testing over RDP. You can place Remote Desktop Canary outside or inside your network, and it will test all the way through the gateway, broker, and session hosts. It can alert you if login times change beyond their average, black screens start occurring, GPOs take longer to process, FSLogix breaks, etc. You can also create login scripts for it that will launch specific programs and then observe whether or not those applications did launch properly, sending you routine emails with screenshots showing you what happened. Finally, it can build diagnostic logs so you can investigate if there are problems that are occurring at a low level in the RDP stack, such as gateway to broker handoffs, broker to session host handoffs, and the like.

From my perspective, the best testing approach for your staging deployment combines both manual and automated testing.

Step 4 – Know the Most Common Post-Update Issues in RDS Deployments, and Test For Them

These are the most common hotspot problems I see that can develop after updates are applied to RDS farms:

• Black screen issues after Windows OS or application updates on session hosts. A lot of times this has to do with things like the AppReadiness service, or other interactions between applications and the Explorer process that can cause problems.
• Application performance issues after updates, where an application starts using more CPU or RAM than previously budgeted for, which decreases per user performance on your RDS hosts. I saw this with a client who updated a PDF reader application, which started using a lot more background CPU. By the way, our Remote Desktop Commander Suite product is helpful in pinpointing these performance issues before and after updates.
• FSLogix/UPD problems and/or problems with Windows Search. Sometimes Windows Search can interfere with user profile technologies like User Profile Disks and FSLogix, which can prevent profiles from being unloaded, and can lead to temporary profiles being created, black screens, and other problems. In other cases, there are simply standalone bugs introduced to UPDs or FSLogix after an update.

A Recap On How To Update Remote Desktop Services Smoothly

To summarize, here are the things you need to do to insulate yourself from breaking changes in your RDS deployments post Windows OS updates or third party updates:

• Set up a separate RDS staging deployment in the same domain, but a different OU, with the same apps installed on it as your production deployment.
• Use Windows Server Update Services (WSUS) or equivalent to “audition” application and OS updates on the RDS staging deployment servers first. For third party application updates, also install them on the staging deployment servers first.
• Perform a combination of manual and automated login testing regularly to catch update-related problems in the staging deployment before installing those updates on your production RDS deployment.