As some of you may know Orb Data run a remote managed service in which we offer to fully manage an aspect of an Information Technology Service Management (ITSM) solution such as Monitoring, Scheduling or Mobile Device Management. A facet of what we deliver as part of this service is  to patch the solution as needed which in our marketing could be just a single bullet point but in reality can be a large part of the workload.  It is also a part of the management of the solution that is often forgotten by customers who may prefer to stay on the initial installed release until something forces them to patch the whole solution at once. However, in our offering we prefer a more proactive approach and as we have performed this for customers for over 5 years we have developed a set of “Orb Data” best practices. This article will share some of these guidelines and also a few pointers to look out for if you are planning an upgrade project yourself.

Orb Data Remote Managed Service Best Practices

We split the patching service into 2 phases:

Ongoing Analysis

There are 4 actions we perform on an ongoing basis regardless of whether we are in a current patch cycle.

Create an inventory of all the items in the estate – clearly the first thing to do is to understand the current state of the environment you wish to patch and therefore the scale of work ahead of you. If you have tools such as TADDM or BigFix to help with this analysis then all the better but if not the information must be collected manually. The inventory should include the following information for each system:

• Operating System and version – This is important for 2 reasons. Firstly, the target patch version may not work on the current version of the operating system and secondly the operating system itself may be out of support. In this case many vendors will not support software that is running on legacy OS versions and so your software is likely to be running on an unsupported configuration already.
• Function of the asset – what function does it deliver to the business (e.g. Primary ObjectServer)
• Installed Software, installation directories and current version – What software is installed on each system? For instance, if you have Netcool which component is running on each system and what is the current version.
• Dependencies – if a piece of software has a dependency (e.g., Java) what version does this need to be? This helps us to know that when we perform an upgrade we also need to ensure that we take account of the documented dependencies.

Collect Vendor information on new releases and patches – Analyse the latest versions that are available from the vendor. IBM have a subscription service to get notified of new patches and releases. For each new patch you will need to analyse:

• Is it needed now? Is it a potential security fix or an issue that you are already having?
• What dependencies does the patch have? There may be a requirement for software such as databases, supporting software or browsers to be upgraded before the underlying software is upgraded. For instance, do you need to upgrade DB2 or java?

For instance, we were recently analysing the containerised version Netcool Operations Insight (NOI) 1.6.2 and looked at the browser support for each part of the functionality. As you can see even for this there was not one browser that was supported for each component but for the record IE v11 and Firefox ESR v45 are your best bets.

Does the operating system of the system support the new patch? For us this is quite important as although we support the software we do not often support the underlying operating system and a patch to this requires actions from the customer which may then extend the timescale of the patch schedule.

In the example below is of a three-way comparison of ITM6 component support for DB2 plus ITM6 and DB2 RHEL support. ITM 6.3.0.7 is the target version, but we needed to work out what platform and what DB2 version is the correct choice.

We may also need to check that the new release can be installed onto the existing hardware. This may be as simple as checking diskspace but could involve checking memory and CPU capacity.

Does the deployment of this patch need to be made on the whole estate? If not which systems need the new release and will they be compatible with the other components (which may now be on an older version) once they have been upgraded?

Create Differential Reports – These are regular reports summarising the latest fix packs /service packs and the key feature enhancements, bug fixes and security updates. Within our service these reports are shared with customers and we find multiple benefits to these:

• Opens the communications from our team to the customer for a new patching cycle
• Ensures the customers are aware of new features and hence can adopt those into their BAU process
• Highlights security issues which assist with prioritisation of the patching cycles

Create/Modify Test Plan – The test plan is a formal document with extensive tests completed only in the Development/Pre-production environment to prove that the patching does not break any of the core product functionality or the operational processes employed by the customer. This may take hours/days to complete. This document will generally be written as part of the first patch cycle and should not need to be redefined each time. Testing that the service is maintained after the upgrade is not dependent on the nature of the patching and is something that should be performed independently of the patching. The test plan should therefore be maintained independently of patching cycles and not re-defined as part of each patching cycle. As the customer develops their solution, uses more services or retires services, then the test plan may expand/contract.

Patch Cycle

Planning

Create a High-Level Plan – Develop a plan to decide what patches should be deployed to which system and in what order. At this stage you don’t need a high level of detail and it should just designate which patches need to be deployed and to which systems, showing if there are any hard end dates that need to be met. For instance, does an Operating System or a piece of software need to be upgraded because it is out of support. Are there urgent security patches that need to be deployed? This is not a static document but should be modified on a regular basis as new information comes in.

Create an Upgrade Plan – This document will include a list of all patches to be deployed in a patching cycle. This will include the patch dependencies and estimates of how long each patch will take to install. Particular mention should be on dependencies that are outside of the core software you are planning to upgrade as this may involve different teams (such as databases, java, operating system versions and libraries). The document should include the current version, the suggested version and any upgrade path in between. The document should also mention the advantages of the patch and any security fixes that it provides. This document then has to get agreement from the customer or internal approval if this is being done directly. At Orb Data we use a Kanban board in our service desk to assign each of the tasks to our various staff. This gives us great visibility as to what has been done and what still needs actioning.

We have also found that we should allow multiple people to review the plan rather than relying on a single person. It is easy to miss something but the more reviewers we use the less likely this becomes.

Test in Sandpit

Test the patch – The first step for us is always to test the patch in a Sandpit environment that as closely mirrors our customer’s installation as possible. This not only allows us to make sure the patch works but also means that we can document the process step by step in order to create a technical deployment document so when we perform the upgrade on the customer’s site we minimise the risk of the change and the time of the outage. This document should contain some basic verification tests to confirm functionality after patching and optionally at key check-points during the patching .

The upgrade time is also recorded so that when the change request is created by the customer they will have a good idea of how long the upgrade will take and how much downtime needs to be planned for. In this stage we check the test and verification plan which should have already been created.

Lastly, we must also include a roll-back plan to quickly reverse patches and return the system to a pre-patched state if significant issues in performance or functionality are encountered after patch deployment. Many of our customer’s estates are virtualised and so as part of the process we may take a snapshot that will allow us to roll back to a stage before the patch was deployed. In other cases, we perform backups or test the vendor’s rollback procedure. A multi-level backup and rollback plan may be reliant on local filesystem backups, application rollback options, virtual image snapshots and filesystem backup tools, for example TSM.

Deploy to Development/Pre-Production

Schedule deployment – Once our initial testing phase is done we are able to agree with the customer a schedule that is mutually convenient. We prefer if we can first deploy in development, then pre-production before finally deploying in production.  This may need a change to be approved before we can proceed but the documentation we have provided has specifically been created to allow for this process to be relatively easy. Some customers require a change request for Pre-production, others do not. All require an agreed date. Sometimes these patch deployments will be small and on other occasions could be a large exercise that could include a new operating system or even a new server (although this will generally fall under a migration project rather than an upgrade).

Deploy Patches – If the customer has a development environment we will initially execute the implementation plan there, if not we will deploy in pre-production. If there are changes found from the sandpit deployment we will create a 2^nd draft of implementation plan. Post deployment we will execute complete the Test plan (extensive tests that may take a day or 2 to complete) to prove that the deployment was successful. If not we will roll-back and tune the implementation plan. When the deployment is successful we will create a final version of Implementation plan for production and then schedule the production patching.

Deploy to Production

Production Deployment – The primary goal for Orb Data during the production deployment is to perform the documented procedure at the prescribed time and with minimal disruption to the production service. By this point the actions we will take and the time we are expecting for any downtime will have been agreed and so it will be just a case of notifying the customer that we are starting. In some cases, we will communicate with customer’s staff (through Slack for instance) as we are performing the patch to inform them of progress. Key checkpoints will be included into larger patching cycles to verify the success of intermediate steps before continuing.

Verify the production patch deployment – Once we patch the production environment we will complete the verification steps (a subset of the full test plan) to formally verify that all planned deployments have worked. It is important that these verification steps cover the key operational processes. If significant issues were encountered, the roll-back plan will be implemented to return the system to the pre-deployment state.

Why use the Orb Data Patching Service?

Back at the start of this blog I commented that many customers leave patching their environment until there is a specific requirement (i.e., a bug, an incompatible operating system or a security issue) as they simply don’t have the time to invest in a proactive patching approach. Ultimately though the software will need to be patched and leaving the estate unpatched will inevitably lead to a large project and a lot of disruption.  For this reason, we are now offering the patching service without the need to take out Orb Data’s larger managed service. Our remote patching service will take away the potential worry and downtime associated with a large project as we will proactively patch your systems throughout the year.

Before 2020 an objection we may have heard from a customer was that they prefer on-site resource, but, if the last year has taught us anything it’s that almost every IT job can be performed remotely. I wrote a blog on this subject called Evolving the use of Remote Resource after Covid-19 which you can read here.

If you would like a quote for your environment or simply discuss your patching needs then please email me at simon.barnes@orb-data.com

Visits: 446