As stated in my post “Orchestrator 2012: Patch a server with SCCM 2012” we had a request to reboot and patch groups of servers in parallel. The requirements were: Restart servers from different groups parallel, manual or scheduled start, do not go on with the rest of the servers in a group if one fails in this group.
How can we do that? First of all: use System Center Orchestrator 2012 – the automation tool from Microsoft.
Then I use SQL to provide the server names and store the status of the process.
I have a OrchestratorTemp database with two tables in there (plus the table described for the patching – see my blog):
The ServerStatus table has some entries already filled, when the runbook starts:
Servername, Grouping.
The Grouping had the values “OWA” and “General”. So servers of these two groups can be rebooted in parallel.
The start workflow looks like this:
It has to be started with the following parameter: Patch = Yes/No. This defines if patches should be applied or not.
If you need to schedule the reboots then you can add a schedule runbook in front of it which checks the date and initializes this runbook with the required start parameter.
It initiates the “Start groups” runbook and waits for completion. After the reboots it checks the patch status, checks the overall status and empties the tables (in the server status table it only deletes the fields which show the status).
Start Groups
This runbook enables the parallelity and can be extended with more groups.
The “Get Server Groups” activity runs the following query: Select Distinct Grouping from dbo.ServerStatus.
The output will be used to start the “Control” runbook and fill the parameter “Grouping”. The parameter “Patch” is also provided to the sub runbook.
This runbook helps to ensure that the rest of the servers in a group are skipped if one server fails.
It has a “Job concurrency” of the number of groups.
Details:
- “Get failed Server in Group” : Select Servername from dbo.ServerStatus where Grouping =’%Grouping%’ AND Status = ‘Failed’
- “Get next Server in Group” : Select Top 1 Servername from dbo.ServerStatus where Grouping =’%Grouping%’ AND Status is NULL.
- When it found a server then it initiates the “Maintenance” runbook with the parameters: “Servername”,”Patch” and “Group”. It waits for completion.
Maintenance
This is the main reboot runbook. It could be split up to multiple sub runbooks, but I only took the patching part out of it. It can also be extended with pre or post activities to stop services or do other tasks around the reboot.
This runbook has a “Job concurrency” of the number of parallel groups.
The easiest way to follow this workflow is to go straight from top to down. The enties in on the left and right are only for logging.
The main things this workflow is doing are: ping the computer, start SCOM maintenance mode, install patches, reboot, check netlogon service to see that the system is up, check patch status, check services and restart if necessary, check general service status, stop maintenance. Additionally it logs the status of the steps and sends out emails.
Here are the details of the non standard activities:
- Get NetLogon Service Status: This is a “Run Command” activity running on the local on the runbook server. sc \\%Servername% query netlogon
- Get Citrix Services: This is a “Run .Net” activity to get application specific services – here for Citrix. It runs a PowerShell script and publishes the variable “output” :
$output=@()
$services = get-wmiobject win32_service -computername %Servername% | where {($_.displayname -like ‘*Citrix*’) -and ($_.Startmode -eq ‘Auto’)}
foreach ($service in $services)
{
$output+=$service.displayname
} - The Get-FQDN activity is described here.
Neil Peterson has written also a complex runbook about patching a Hyper-V cluster. He used some other methods to intialize the patching and presented the whole process on the MMS2013. You can get the details and watch the session here: http://blogs.technet.com/b/neilp/archive/2013/04/15/mms2013-session-now-on-channel-9-patching-a-hyper-v-cluster-with-orchestrator-configuration-manager-including-downloadable-runbook-exports.aspx
The Runbooks can be downloaded here.