This week I realized that I’m seeing a problem repeat itself over time and organizations. Today I realized that there is a broader application to the problem, and I’m beginning to think that there may be a solution to the problem. I say Supervision, again because I believe that automation without supervision is doomed to fail. In the end, supervision is a major part of the solution to this problem as well.

I have watched as companies deploy computerized control systems using traditional project methodologies and the local maintenance personnel or users take responsibility for the administration of these systems once the projects end. The degree and quality of administration varies with the people who do it and the amount of time that they are willing to spend on this new and additional responsibility. Most of the time this includes physical inventory, backups and maybe anti-virus software, but only rarely does it include patch management, recovery exercises, or performance monitoring. The company makes a capital investment (the project), reaps the benefits of the new automated controls (return on investment), and dodges ongoing maintenance costs by hiding them in existing maintenance infrastructure or failing to perform them entirely.

More recently a central authority from elsewhere offers to provide administrative services to the system in the name of compliance or security. They have a project (another capital investment) to install centralized tools such as backup software, anti-virus software, patch management systems, performance monitoring systems, and sometimes intrusion detection or log management systems. A consequence of the centralization is that all of these systems are administered by the central authority and the local maintenance personnel cannot easily supervise the effectiveness of the tools. In some cases the tools are not visible to the locals, in others the locals are not provided meaningful training about how to reach or operate them, and in yet others the locals are encouraged to spend their time elsewhere and to simply trust the centralized authority to handle the responsiblity of administering the control systems.

In my long career I’ve seen what happens when the centralized authority fails to meet its responsibility for whatever reasons. A switch dies and the locals discover that a spare was never provisioned. A computer fails and we discover that the database wasn’t backed up by the software. A virus infects a computer and is reported to the centralized console, but nobody took action on the alert in time to prevent damage. Even when the central authority does perform some of the tasks it rarely helps to solve the problems that arise; and now the locals are trying to fix a reported problem that they do not understand using skills that they haven’t been trained to have with time that is borrowed from their full-time jobs.

In a similar way I am aware that some Owners hire Infrastructure as a Service (IAAS) providers and Managed Security Service Providers (MSSPs) to provide administrative services and later find that they were billed for services that weren’t performed completely, or were inadequate for their needs. The owner is responsible for their assets from beginning to end and can only delegate authority to others to help manage that responsibility.

In the end, the Owner is responsible for administering their cyber assets. When the Owner delegates authority to others, either internally or externally they are still responsible to ensure that the responsibility is met. If the Owner is not technically capable of supervising their delegates then they can delegate that authority to a third party and still meet the responsibility.

Local maintenance personnel end up in a curious place; when the systems fail they are personally responsible for repairing them. I have rarely seen a CEO or senior manager at the plant on a midnight shift, a weekend, or a holiday when the repairs are ongoing. Perhaps management pays a price in money, but this is significantly different than spending one’s personal time as the price. This harkens back to discussions about why a personally written Thank You card can mean more than an electronically transfered check.

Part of the solution is for the Owner to establish a Program to administer computerized control systems, to staff that Program adequately to meet the requirements, and to provide projects to enable everyone manage the program and any repair events. These projects may be to provide tools or to provide training. Part of the solution is for the Owner to supervise the Program to ensure that it is effective through a combination of audits and exercises.