UR9 for SCOM 2012 R2 – Step by Step

This is an updated article replacing the original – to include the deployment of the Linux MP's which shipped later.  Since Microsoft changed blog platforms over to WordPress – it will not allow me to update the previous one.

NOTE:  I get this question every time we release an update rollup:   ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2012R2 and never applied an update rollup – you can go strait to the latest one available.  If you applied an older one (such as UR3) you can always go straight to the latest one!

KB Article for OpsMgr:  https://support.microsoft.com/en-us/kb/3129774

KB article for Linux updates:  https://support.microsoft.com/en-us/kb/3141435

Download catalog site:  http://catalog.update.microsoft.com/v7/site/Search.aspx?q=3129774

Key fixes:

SharePoint workflows fail with an access violation under APM A certain sequence of the events may trigger an access violation in APM code when it tries to read data from the cache during the Application Domain unload. This fix resolves this kind of behavior.

Application Pool worker process crashes under APM with heap corruption During the Application Domain unload two threads might try to dispose of the same memory block leading to DOUBLE FREE heap corruption. This fix makes sure that memory is disposed of only one time.

Some Application Pool worker processes become unresponsive if many applications are started under APM at the same timeMicrosoft Monitoring Agent APM service has a critical section around WMI queries it performs. If a WMI query takes a long time to complete, many worker processes are waiting for the active one to complete the call. Those application pools may become unresponsive, depending on the wait duration. This fix eliminates the need in WMI query and significantly improves the performance of this code path.

MOMAgent cannot validate RunAs Account if only RODC is availableIf there's a read-only domain controller (RODC), the MonAgent cannot validate the RunAs account. This fix resolves this issue.

Missing event monitor does not warn within the specified time range in SCOM 2012 R2 the first time after restartWhen you create a monitor for a missed event, the first alert takes twice the amount of time specified time in the monitor. This fix resolves the issue, and the alert is generated in the time specified.

SCOM cannot verify the User Account / Password expiration date if it is set by using Password Setting objectFine grained password policies are stored in a different container from the user object container in Active Directory. This fix resolves the problems in computing resultant set of policy (RSOP) from these containers for a user object.

SLO Detail report displays histogram incorrectlyIn some specific scenarios, the representation of the downtime graph is not displayed correctly. This fix resolves this kind of behavior.

APM support for IIS 10 and Windows Server 2016Support of IIS 10 on Windows Server 2016 is added for the APM feature in System Center 2012 R2 Operations Manager. An additional management pack Microsoft.SystemCenter.Apm.Web.IIS10.mp is required to enable this functionality. This management pack is located in %SystemDrive%\Program Files\System Center 2012 R2\Operations Manager\Server\Management Packs for Update Rollups alongside its dependencies after the installation of Update Rollup 9. Important Note One dependency is not included in Update Rollup 9 and should be downloaded separately:

Microsoft.Windows.InternetInformationServices.2016.mp

APM Agent Modules workflow fail during workflow shutdown with Null Reference ExceptionThe Dispose() method of Retry Manager of APM connection workflow is executed two times during the module shutdown. The second try to execute this Dispose() method may cause a Null Reference Exception. This fix makes sure that the Dispose() method can be safely executed one or more times.

AEM Data fills up SCOM Operational database and is never groomed outIf you use SCOM's Agentless Exception Monitoring to examine application crash data and report on it, the data never grooms out of the SCOM Operational database. The problem with this is that soon the SCOM environment will be overloaded with all the instances and relationships of the applications, error groups, and Windows-based computers, all which are hosted by the management servers. This fix resolves this issue. Additionally, the following management pack's must be imported in the following order:

Microsoft.SystemCenter.ClientMonitoring.Library.mp

Microsoft.SystemCenter.DataWarehouse.Report.Library.mp

Microsoft.SystemCenter.ClientMonitoring.Views.Internal.mp

Microsoft.SystemCenter.ClientMonitoring.Internal.mp

The DownTime report from the Availability report does not handle the Business Hours settingsIn the downtime report, the downtime table was not considering the business hours. This fix resolves this issue and business hours will be shown based on the specified business hour values. The updated RDL files are located in the following location:

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Reporting

To update the RDL file, follow these steps:

Go to http://MachineName/Reports_INSTANCE1/Pages/Folder.aspxMachineName //Reporting Server.

On this page, go to the folder to which you want to add the RDL file. In this case, click Microsoft.SystemCenter.DataWarehouse.Report.Library.

Upload the new RDL files by clicking the upload button at the top. For more information, see https://msdn.microsoft.com/en-us/library/ms157332.aspx.

Adding a decimal sign in an SLT Collection Rule SLO in the ENU Console on a non-ENU OS does not workYou run the System Center 2012 R2 Operations Manager Console in English on a computer that has the language settings configured to use a non-English (United States) language that uses a comma (,) as the decimal sign instead of a period (.). When you try to create Service Level Tracking, and you want to add a Collection Rule SLO, the value you enter as the threshold cannot be configured by using a decimal sign. This fix resolves the issue.

SCOM Agent issue while logging Operations Management Suite (OMS) communication failureAn issue occurs when OMS communication failures are logged. This fix resolves this issue.

Issues that are fixed in the UNIX and Linux management packs

Discovery of Linux computers may fail for some system locales Using the Discovery Wizard or Windows PowerShell cmdlets to discover Linux computers may fail during the final Agent Verification step for computers that have some system locales, such as zh_TW.UTF-8. The scxadmin command that is used to restart the agent during the discovery process did not correctly handle Unicode text in the standard out-of-the-service command.

The UNIX/Linux Agent intermittently closes connections during TLS handshakingSymptoms include the following:

Failed heartbeats for UNIX or Linux computers, especially when the SSLv3 protocol is disabled on the Management Servers.

Schannel errors in the System log that contain text that resembles the following:

A fatal error occurred while creating an SSL client credentials. The internal error state is 10013.

WS-Management errors in the event log that contain text that resembles the following:

WSManFault Message = The server certificate on the destination computer (<UNIX/LINUX-COMPUTER-NAME) has the following errors: Encountered an internal error in the SSL library. Error number: -2147012721 0x80072F8F A security error occurred

 

Lets get started.

From reading the KB article – the order of operations is:

Install the update rollup package on the following server infrastructure:

Management servers

Gateway servers

Web console server role computers

Operations console role computers

Apply SQL scripts.

Manually import the management packs.

Update Agents

Now, NORMALLY we need to add another step – if we are using Xplat monitoring – need to update the Linux/Unix MP's and agents.   However, in UR8 and UR9 for SCOM 2012 R2, there are no updates for Linux

1.  Management Servers

Since there is no RMS anymore, it doesn't matter which management server I start with.  There is no need to begin with whomever holds the RMSe role.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 3 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:

Then extract the contents:

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator (SA) role to the database instances that host your OpsMgr databases.

My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update usually does not provide any feedback that it had success or failure. 

I got a prompt to restart:

I choose yes and allow the server to restart to complete the update.

You can check the application log for the MsiInstaller events to show completion:

Log Name:      Application Source:        MsiInstaller Date:          1/27/2016 9:37:28 AM Event ID:      1036 Description: Windows Installer installed an update. Product Name: System Center Operations Manager 2012 Server. Product Version: 7.1.10226.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2012 R2 Operations Manager UR9 Update Patch. Installation success or error status: 0.

You can also spot check a couple DLL files for the file version attribute. 

Next up – run the Web Console update:

This runs much faster.   A quick file spot check:

Lastly – install the console update (make sure your console is closed):

A quick file spot check:

Additional Management Servers:

I now move on to my additional management servers, applying the server update, then the console update and web console update where applicable.

On this next management server, I will use the example of Windows Update as opposed to manually installing the MSP files.  I check online, and make sure that I have configured Windows Update to give me updates for additional products: 

The applicable updates show up under optional – so I tick the boxes and apply these updates.

After a reboot – go back and verify the update was a success by spot checking some file versions like we did above.

Updating Gateways:

I can use Windows Update or manual installation.

The update launches a UI and quickly finishes.

Then I will spot check the DLL's:

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

***NOTE:  You can delete any older UR update files from the \AgentManagement directories.  The UR's do not clean these up and they provide no purpose for being present any longer.

2. Apply the SQL Scripts

In the path on your management servers, where you installed/extracted the update, there are two SQL script files: 

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment of clean install)

First – let's run the script to update the OperationsManager database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file.  Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

Click the "Execute" button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.  I have had customers state this takes from a few minutes to as long as an hour. In MOST cases – you will need to shut down the SDK, Config, and Monitoring Agent (healthservice) on ALL your management servers in order for this to be able to run with success.

You will see the following (or similar) output:

or

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a production environment, you almost certainly have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run.

Technical tidbit:   Even if you previously ran this script in UR1, UR2, UR3, UR4, UR5, UR6, UR7, or UR8, you should run this again for UR9, as the script body can change with updated UR's.

Next, we have a script to run against the warehouse DB.  Do not skip this step under any circumstances.    From:

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment of clean install)

Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file UR_Datawarehouse.sql.  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

If you see a warning about line endings, choose Yes to continue.

Click the "Execute" button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

3. Manually import the management packs

There are 55 management packs in this update!   Most of these we don't need – so read carefully.

The path for these is on your management server, after you have installed the "Server" update:

\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language.  I will remove all the MP's for other languages (keeping only ENU), and I am left with the following:

What NOT to import:

The Advisor MP's are only needed if you are using Microsoft Operations Management Suite cloud service, (Previously known as Advisor, and Operation Insights).

The APM MP's are only needed if you are using the APM feature in SCOM.

Note the APM MP with a red X.  This MP requires the IIS MP's for Windows Server 2016 which are in Technical Preview at the time of this writing.  Only import this if you are using APM *and* you need to monitor Windows Server 2016.  If so, you will need to download and install the technical preview editions of that MP from https://www.microsoft.com/en-us/download/details.aspx?id=48256

The TFS MP bundle is only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc.  If you are not currently using these MP's, there is no need to import or update them.  I'd skip this MP import unless you already have these MP's present in your environment.

However, the Image and Visualization libraries deal with Dashboard updates, and these always need to be updated.

I import all of these shown without issue.

4.  Update Agents

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):  

One the Management servers where I used Windows Update to patch them, their agents did not show up in this list.  Only agents where I manually patched their management server showed up in this list.  FYI.   The experience is NOT the same when using Windows Update vs manual.  If yours don't show up – you can try running the update for that management server again – manually.

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending.

In this case – my agents that were reporting to a management server that was updated using Windows Update – did NOT place agents into pending.  Only the agents reporting to the management server for which I manually executed the patch worked.

I manually re-ran the server MSP file manually on these management servers, from an elevated command prompt, and they all showed up:

 

You can approve these – which will result in a success message once complete:

 

Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:

  5.  Update Unix/Linux MPs and Agents

The current Linux MP's can be downloaded from:

https://www.microsoft.com/en-us/download/details.aspx?id=29696

7.5.1050.0 is current at this time for SCOM 2012 R2 and these shipped shortly after UR9. 

****Note – take GREAT care when downloading – that you select the correct download for SCOM 2012 R2.  You must scroll down in the list and select the MSI for 2012 R2:

Download the MSI and run it.  It will extract the MP's to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\

Update any MP's you are already using.   These are mine for RHEL, SUSE, and the Universal Linux libraries. 

You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports.  Give it plenty of time to complete the process of the import and MPB deployments.

Next – you need to restart the "Microsoft Monitoring Agent" service on any management servers which manage Linux systems.  I don't know why – but my MP's never drop/update in the \Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits filder until this servcie is restarted.

Next up – you would upgrade your agents on the Unix/Linux monitored agents.  You can now do this straight from the console:

 

You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.

Finally:

6.  Update the remaining deployed consoles

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

Known issues:

See the existing list of known issues documented in the KB article.

1.  Many people are reporting that the SQL script is failing to complete when executed.  You should attempt to run this multiple times until it completes without error.  You might need to stop the Exchange correlation engine, stop all the SCOM services on the management servers, and/or bounce the SQL server services in order to get a successful completion in a busy management group.  The errors reported appear as below:

—————————————————— (1 row(s) affected) (1 row(s) affected) Msg 1205, Level 13, State 56, Line 1 Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. Msg 3727, Level 16, State 0, Line 1Could not drop constraint. See previous errors. ——————————————————–

Source: UR9 for SCOM 2012 R2 – Step by Step