Thiago N. Felippe, CEO of Aiqon, explains why CIOs and CISOs are likely to create their own software testing/secure patch management labs.
The recent global outage highlighted two key truths for CIOs, CISOs and the entire C-level: Software-as-a-Service (SaaS) is a reality, with organizations utilizing solutions that are automatically updated (patching) without always running comprehensive tests on these new components.
It also became evident that in the digital economy, companies rely on technology to conduct business, and when a failure of the magnitude experienced on July 19 occurs, a contingency plan is essential to sustain operations. Airlines that managed to operate did so based on paper tickets.
According to DownDetector, on Friday, July 19, it received 311,000 notifications of unstable or down services (B2C and B2B portals), with 58,000 from the US alone. In Paraná, at the Port of Paranaguá, there was a seven-kilometer-long queue of trucks with containers waiting for the system to come back online. This might explain why, according to QR Code Generator, Google searches for alternatives to the Windows platform surged 290% compared to the previous day.
Patching without manual updates
Not all organizations have automated patch management solutions that ensure Windows desktops or servers are updated without manual intervention, machine by machine. Even after the security platform vendor that caused the outage released a fix, only those who had already effectively managed the challenging and routine task of remote and secure patch management were able to recover their machines quickly.
It’s noteworthy that the lockdown of millions of Windows machines worldwide was triggered by a Security-as-a-Service platform. This platform, as per its contract with clients, offers a one-hour SLA to deliver a fix/vaccine against newly detected malware or breaches. Within this model and with the current threat landscape, patch management for this platform is a constant task that should not have caused the issues it did.
On Wednesday, July 24, CrowdStrike revealed a bug in the Falcon Sensor software update. This bug was not identified during the manufacturer’s testing phase, which led to the patch being released and distributed to all clients of the platform.
To prevent future failures like this, many CIOs and CISOs are likely to create their own software testing/secure patch management labs. Instead of continuing to rely on software platforms – including cybersecurity services – they will perform high-performance tests to confirm that platform updates are secure and do not conflict with other technologies. Brazil already has secure patch management solutions that facilitate the setup of these labs. The goal is to accelerate the testing of updates without affecting business processes.
Successful strategies for creating testing labs
- Create a virtual testing environment: Clone critical points of the production environment to ensure that a patch that negatively impacts a specific device is identified and rejected before being distributed across the company’s machines.
- Check for an uninstaller: This is a key factor to consider in any testing environment. If the patch lacks an uninstallation option, additional testing will be necessary.
- Review test results with a separate team: Ensure that the results are reviewed by a team other than the testers themselves.
Aligning software testing and patch management with business needs
Above all, the organization of a corporate software testing lab must follow business logic. Only CIOs and CISOs with a deep understanding of business operations connected to the digital solutions in use can measure the impact of each update on the company. This clarity will help establish testing priorities, correction priorities, and determine when it is best not to deploy a patch within the organization.
The recent outage might represent a turning point in everything related to software testing and secure patch management. There were errors in testing, processes and updates, leading to the collapse of digital operations. In the SaaS era, the bug was distributed via the cloud globally and is still being mitigated, in some cases, machine by machine. In this context, new strategies for software testing and secure patch management are essential.