I was recently tasked by my boss to upgrade our 2012 R2 environment to 2016, amongst other things we’re going to need it for some screenshots and testing later on so we needed to get it done. This is my experience of it and I thought I’d share it with you as it will probably something you are tasked to do in the near future.
So with any upgrade for Ops Mgr. I normally check out the TechNet article that’s published:
So in summary the hi level is before we :
- Check Build Number to ensure we’re at UR9 or above
- Check Sequencing Order for other components for System Center
- Upgrading a distributed management group – order of execution
Now I’m doing this Upgrade out of order as we need to get this done but in general you would need to upgrade the existing components of System Center in the environment first and that would be:
- Configuration Manager
- Data Protection Manager
- Operations Manager
- Virtual Machine Manager
First off – check your UR level of Operations Manager – it should be at least UR9. Previous blogs have covered this such as <Link to Kev Holman’s Blog here> but essentially you need to have a build number of 1177 or greater.
A really good resource for build numbers is:
Essentially you’re looking for at least a file version of 1177 on the core files in the following directory on your Management servers:
%Installation Directory Path%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server
Order of Execution for Upgrading
If you have done any sort of UR Upgrades before you will notice that this is more similar to that procedure than what we saw when upgrading from 2007 R2.
High level overview of System Center 2016 Operations Manager upgrade steps for a distributed management group
The following steps outline the process for upgrading a distributed management group:
- Accomplish Pre-Upgrade Tasks
- Upgrade the initial management server and then additional management servers (each management server must be upgraded)
- Upgrade ACS (because the ACS server must be on same machine as a management server, we recommend you perform this step along with the upgrade of the management server on which ACS resides.)
- *Upgrade Gateway(s)
- Upgrade Console
- Push install to agent(s) / upgrade manually installed agents
- Upgrade Web Console
- Upgrade Reporting Server
- Perform Post-Upgrade Tasks
*Steps 4 through 8 can be performed in parallel after all management servers have been upgraded.
The following actions are required and I tend to create a checklist to mark what I’ve done 🙂
- Review the Operations Manager Event Logs
- Errors found – investigating
- Clean-up the Database (ETL Table)
- Configure agents to failover between multiple gateway servers so all agents reporting to a gateway have a failover gateway assigned.
- Remove Agents from Pending Management
- Disable Notification Subscriptions
- Disable any connectors
- Stop the Microsoft Monitoring Agent, System Center Data Access Service, System Center Configuration Management and Microsoft Monitoring Agent services on all management servers except the one being upgraded
- Verify that the Operational Database Has More Than 50 Percept Free Space
- Back up the Operations Manager Databases
- Update the agent’s health service cache size temporarily to prevent loss of data while Management and Gateway servers are upgraded.
- Disable the Operations Manager website in IIS.
- Review the Operations Manager Event Logs
Clean-up the Database (ETL Table)
So from the TechNet article – there is an instruction to clean up the ETL table on the OpsManager Database so opened up a SQL Management Studio session and ran the following script:
— (c) Copyright 2004-2006 Microsoft Corporation, All Rights Reserved —
— Proprietary and confidential to Microsoft Corporation —
— File: CatchupETLGrooming.sql —
— Contents: A bug in the ETL grooming code could have left the customer —
— Database with a large amount of ETL rows to groom. This script will groom —
— The ETL entries in a loop 100K rows at a time to avoid filling up the —
— Transaction log —
DECLARE @RowCount int = 1;
DECLARE @BatchSize int = 100000;
DECLARE @SubscriptionWatermark bigint = 0;
DECLARE @LastErr int;
— Delete rows from the EntityTransactionLog. We delete the rows with TransactionLogId that aren’t being
— used anymore by the EntityChangeLog table and by the RelatedEntityChangeLog table.
SELECT @SubscriptionWatermark = dbo.fn_GetEntityChangeLogGroomingWatermark();
WHILE(@RowCount > 0)
DELETE TOP(@BatchSize) ETL
FROM EntityTransactionLog ETL
WHERE NOT EXISTS (SELECT 1 FROM EntityChangeLog ECL WHERE ECL.EntityTransactionLogId = ETL.EntityTransactionLogId) AND NOT EXISTS (SELECT 1 FROM RelatedEntityChangeLog RECL
WHERE RECL.EntityTransactionLogId = ETL.EntityTransactionLogId)
AND ETL.EntityTransactionLogId < @SubscriptionWatermark;
SELECT @LastErr = @@ERROR, @RowCount = @@ROWCOUNT;
Ran the above script and got the following output:
Configure agents to failover between multiple gateway servers so all agents reporting to a gateway have a failover gateway assigned.
N/A in our environment
Remove Agents from Pending Management
None in our environment
Disable Notification Subscriptions
None in environment
Disable any connectors
Disabled the Connectors from the Service Manager Perspective
Stop the Microsoft Monitoring Agent, System Center Data Access Service, System Center Configuration Management and Microsoft Monitoring Agent services on all management servers except the one being upgraded
So I did this across all the Management Servers (5 in our environment – will be reducing this at some point as not needed any more!)
This step is required so there are no calls to either database whilst it is being upgraded.
Verify that the Operational Database Has More Than 50 Percept Free Space
This should be a pretty common exercise for anyone who has done a UR before but if not again within your SQL Management Studio session, right click on the Operations Manager Database, choose Reports –> Disk Usage and the following Report is generated:
Standard SQL Size Report for the OpsDB
Also checked the DW DB
As you can see All fine 🙂
Back up the Operations Manager Databases
Update the agent’s health service cache size temporarily to prevent loss of data while Management and Gateway servers are upgraded.
Not undertaking due to type of environment & amount of data – if this was a full size production environment I would potentially still skip as the amount of down time the Management Group is down for is minimal (about 1 – 2 hours at most)
Disable the Operations Manager website in IIS.
Disabling the Website
Logged onto my main Management Server (Running the RMSe role)
Ran the install media & downloaded the UR1 Update files
Following Splash screen appeared
Accept the Terms and sign your life away:
Choose the appropriate Installation Location:
The next screen shows us we have a new requirement around the report viewer controls which is Report Viewer Runtime 2015. This in turn also has a dependency on SQL CLR Runtime 2014 so both will be required to be downloaded
SQL CLR 2014 Runtime hides within the SQL Server Feature Packs, 2014 edition!
Click download as usual:
Scroll down until you find “SQLSysClrTypes.msi – yes we do need the 64 bit version 🙂
Save it, run it and install it.
You won’t get prompted for Install paths etc. as the existing SQL components dictate where this install will end up.
Then do the same for the Report Viewer Runtime 2015
Click “Verify Prerequisites Again”
Enter the SAME Credentials as the Existing DAS Account otherwise you will have serious issues later on:
Confirm the details are correct and then run the upgrade:
A very familiar screen will appear:
And should hopefully end like this:
But mine ended up like this:
Okay – I thought we were running full fat here ?
No bother – I’ll check that out later – running through each of our additional Management Servers now:
This one has the Web Console Role on and is updated automatically:
No ACS or Gateways in our environment so skipping those for the purposes of this install 🙂
Reporting Server – Need to Await the Management Server that it reports to to be upgraded first
Got this error:
Now the server is updated but not currently online – figuring this is the issue – all the servers needed a restart anyway 🙂
Still no joy, restarted the Reporting Server – No joy either.
Google became my pal when the following forum turned up a suggestion
Makes sense – the binaries cant connect to check the running data access service???
Rerunning it gives the following
Good – install the Report Viewer & SQL CLR and should be good to go 🙂
This installed and thus completed the install giving us a beautiful 2016 environment with Maintenance Mode, Alert Tuning, Network extensible monitoring …..
I then took the opportunity to upgrade to UR1 which requires the normal UR upgrade procedure. Lots of people have already blogged about that and the man himself Mr Holman always does a first rate blog and step by step which you can find here:
And the official Support and Catalog articles for UR1:
All left to do at that point was to re-enable the connectors, subscriptions and check the system over and we have a 2016 Operations Environment ready for use 🙂
Hope that helps anyone planning or doing it with the errors I came across and if you need to get in touch
Till next time, happy SCOMMING.