Segmentation of Services

I’ve read and heard recommendations of network segmentation for years, and have made my living segmenting networks from time to time. Recently I’ve realized that segmenting the network services are just a subset of segmenting other services for the system.

We’ve watched for a while as a centralized Active Directory service is compromised and then used to propogate malware throughout the domain. The same bad actors sometimes target backup (or recovery) systems; if a backup system provides recovery services for a large or critical group of systems then it itself becomes a significant point of failure.

In industrial control systems one of the unacknowledged strengths has been the distributed (or segmented) nature of the systems, as they have traditionally been deployed without taking ‘advantage’ of centralized corporate Information Technology (IT) services. It seems that there is now a push to provide centralized management of Operational Technology (OT) systems for all kinds of useful things. Inventory management, vulnerability management, malicious code protection (anti-virus, whitelisting, or intrusion detection), log management, performance management, and so on. In many cases these services aren’t implemented well or at all in the OT environments and so the centralized provision of these services seems like an easy improvement. The local facilities are enticed into participation because the cost is often borne by the greater corporation and not directly by the local budgets.

To what degree should these services be segmented across the enterprise? It doesn’t make sense to have a standalone solution for each service at every one of hundreds of sites; but it also doesn’t make sense to have a single solution for all services for the entire enterprise either. The challenge is to define the balance between the cost to maintain (unacceptable with many small site solutions) versus the cost of compromise (unacceptable with a single enterprise solution).

One approach is to evaluate the business by functional blocks, from an OT perspective. A business segment can be defined as the group of facilities that are required to provide a product or service to the customer; the impact of a failure could be limited to a product, service, or group of similar products or services. Another approach is to define groups of facilities and to plan for the failure of any one group of facilities. The other groups should be able to increase capacity or contribute inventory to cover the failure.

Network segmentation tends to be at two levels. Segmentation of the OT environments from the IT environments, and segmentation of the OT networks within a facility. This provides a series of network bulkheads as in a ship where the failure of one compartment does not cause the entire ship to sink. What is the necessary balance to achieve the same benefit to risk reduction for other services?

Supervision, again

This week I realized that I’m seeing a problem repeat itself over time and organizations. Today I realized that there is a broader application to the problem, and I’m beginning to think that there may be a solution to the problem. I say Supervision, again because I believe that automation without supervision is doomed to fail. In the end, supervision is a major part of the solution to this problem as well.

I have watched as companies deploy computerized control systems using traditional project methodologies and the local maintenance personnel or users take responsibility for the administration of these systems once the projects end. The degree and quality of administration varies with the people who do it and the amount of time that they are willing to spend on this new and additional responsibility. Most of the time this includes physical inventory, backups and maybe anti-virus software, but only rarely does it include patch management, recovery exercises, or performance monitoring. The company makes a capital investment (the project), reaps the benefits of the new automated controls (return on investment), and dodges ongoing maintenance costs by hiding them in existing maintenance infrastructure or failing to perform them entirely.

More recently a central authority from elsewhere offers to provide administrative services to the system in the name of compliance or security. They have a project (another capital investment) to install centralized tools such as backup software, anti-virus software, patch management systems, performance monitoring systems, and sometimes intrusion detection or log management systems. A consequence of the centralization is that all of these systems are administered by the central authority and the local maintenance personnel cannot easily supervise the effectiveness of the tools. In some cases the tools are not visible to the locals, in others the locals are not provided meaningful training about how to reach or operate them, and in yet others the locals are encouraged to spend their time elsewhere and to simply trust the centralized authority to handle the responsiblity of administering the control systems.

In my long career I’ve seen what happens when the centralized authority fails to meet its responsibility for whatever reasons. A switch dies and the locals discover that a spare was never provisioned. A computer fails and we discover that the database wasn’t backed up by the software. A virus infects a computer and is reported to the centralized console, but nobody took action on the alert in time to prevent damage. Even when the central authority does perform some of the tasks it rarely helps to solve the problems that arise; and now the locals are trying to fix a reported problem that they do not understand using skills that they haven’t been trained to have with time that is borrowed from their full-time jobs.

In a similar way I am aware that some Owners hire Infrastructure as a Service (IAAS) providers and Managed Security Service Providers (MSSPs) to provide administrative services and later find that they were billed for services that weren’t performed completely, or were inadequate for their needs. The owner is responsible for their assets from beginning to end and can only delegate authority to others to help manage that responsibility.

In the end, the Owner is responsible for administering their cyber assets. When the Owner delegates authority to others, either internally or externally they are still responsible to ensure that the responsibility is met. If the Owner is not technically capable of supervising their delegates then they can delegate that authority to a third party and still meet the responsibility.

Local maintenance personnel end up in a curious place; when the systems fail they are personally responsible for repairing them. I have rarely seen a CEO or senior manager at the plant on a midnight shift, a weekend, or a holiday when the repairs are ongoing. Perhaps management pays a price in money, but this is significantly different than spending one’s personal time as the price. This harkens back to discussions about why a personally written Thank You card can mean more than an electronically transfered check.

Part of the solution is for the Owner to establish a Program to administer computerized control systems, to staff that Program adequately to meet the requirements, and to provide projects to enable everyone manage the program and any repair events. These projects may be to provide tools or to provide training. Part of the solution is for the Owner to supervise the Program to ensure that it is effective through a combination of audits and exercises.

Thotcon 0x9

Last week I attended Thotcon 0x9 in Chicago and heard a few presentations that made me think. The first was the keynote by Cory Doctorow entitled “The war on general purpose computing is an existential threat to infosec – and the world!”. He made an excellent case for the work of the Electronic Frontier Foundation (EFF) and has changed my opinion from one of little respect to one of strong need for their efforts.

The second was by Wendy Nather entitled “Denial of Trust: The New Attacks”. During this speech she raised the idea that companies (or corporations) consider the loss of data to be akin to Acts of God. This reminded me of a time when some people banded together to fight the deaths of 30,000 Americans per year due to boiler explosions – which the manufacturers attributed to Acts of God. Their efforts formed the American Society of Mechanical Engineers and ultimately led to the National Boiler Code and the licensure of Professional Engineers in the United States.

The third presentation by Karen Elazari was titled “Hackers: Still the Internet’s Immune System?” She makes the excellent point that hackers may be the most effective group opposing the domination of the Internet, collection and access to Big Data, and manipulation of the People today. At some point I heard the question of who will hold governments and companies to account if not hackers? The follow up becomes whether hackers are willing and able to organize in some way in order to maintain their effectiveness.

Lastly, “the Hacker Community Must Always Exist” by Chris Wysopal gave an enlightening history of the adversarial relationship between hackers and industry. He questioned the continued effectiveness of this relationship as technologies such as medical devices and blockchains are deployed, and whether there can be enough hackers to keep up with new technologies.

Altogether these speakers have motivated me to become a better Security Engineer and to make some effort to help find the answers. I see many parallels in the development and use of steam power 140 years ago and our problems in technology today. The scale is huge, but if enough people work together I believe that the answers can be found.

The meaning of IP addresses versus host names

An article in the November 2017 issue of The Internet Protocol Journal raises the excellent point that IP addresses have been replaced by host names in defining coherence to a network.

This week I’ve discovered that IP version 6 reserves an address space that is intended for local communications, normally within a site. This space is defined in RFC 4193, and goes on to explicitly state a design objective: the addresses are not designed to aggregate.

Our IT brethren are being driven away from aggregating addresses by the rise of mobile devices and the internet of things. Within the commercial IT world, host names are replacing addresses in allowing technicians to find specific assets, troubleshoot particular problems, and maintain environmental awareness. Our controls IT world is still working with aggregated addresses and organizing our systems using the addresses.

If the controls environments continue to use Commercial Off-The-Shelf components and to rely on Corporate IT for design and high-level support we need to begin migrating our designs from system organizations built around addressing and shift to comprehensive naming and name resolution services.

Hard Rights

Recently I’ve been amazed at the people who want to do things differently because it is easier.  At a recent campout a leader of our Boy Scout troop came to me and said “we’ve talked it over and had a great idea.  We should go home this evening and not stay the night.”  It was a hot day, and we had to make a conscious effort to be cheerful and have fun.  The comment struck me as a way to make our lives easier.  Following this logic, it would be easier not to go camping at all, but we went camping to accomplish something;  not going means that something will never be realized.  As I told the scouts at our next troop meeting, part of scouting is to learn to take anything that nature can throw at you so that you build the confidence to take anything that life throws at you.

On my drive in this morning I heard a report on NPR discussing why President Trump likes to appoint veteran generals to his staff.  During the story Robert McDonald said the following about the West Point Cadet Prayer, which pretty well sums it up:

“Those words are, ‘Help me to choose the harder right rather than the easier wrong.’ And it’s remarkable, but in business as in life, the easier thing is usually the wrong thing to do,”

http://www.npr.org/2017/08/23/545289536/why-donald-trump-likes-to-surround-himself-with-generals