Big (Sensor) Data

A manufacturing client want to understand how its equipment was performing and how its buildings were being used. With twelve sites and hundreds of pieces of equipment, this was going to be interesting.

The company wanted to understand more about what its sites and equipment were doing. Production line hardware already had some sensor data available, a door access system could tell who was using what building, and environmental control systems could tell when someone was in an office. They wanted a series of dashboards that could tell them in real time what’s going on, from different viewpoints.

Starting with each system, we looked into what data was available, where it was stored, and what each can report upon. Equipment monitoring systems appeared to be well connected, but the data itself was only being used to alert when a motor ran hot, a power supply regulated out of tolerance, or a malfunction. We made a list of all the vendors to contact them later. Environmental system monitoring wasn’t connected and used its own network, with no way of moving the data to the corporate network. It was, however, a rich system with the capability to add further sensors if desired. Door access was well logged but not stored centrally.

We were still without a complete set of dashboards needed, therefore we were flying blind for a while as to the data needed. To have a rich reporting service, we needed to get all the data moved into a central location. We contacted the vendors of the production line equipment to discover what each was capable of providing. We were pleasantly surprised to learn we could have a lot of data collected with little configuration change and no risk to equipment performance.

Naturally, we needed to get the data to a central location and while we recommended the cloud (AWS Redshift in fact) the client’s security group decided due to the ultra-sensitive nature of the resulting dataset, an on-premises data warehouse was the only option. We resorted to using the internal development teams to provide it. In the meantime, we set about solving the problem of the air-gapped door access network.

The organisation maintained an air-gapped network for resilience and security reasons. We respected this and designed a one-way gateway, which wasn’t a firewall and protected the air-gapped network.

The vendors adjusted the equipment logging and we began populating the data warehouse with logging data. As soon as the one-way gateway was ready, door access data began flowing. The environmental services proved a little problematic as logging was sent to a database instead of a flat text file. This problem was overcome by running periodic tasks against the data and extracting the logs. It wasn’t as simple as that as there were other complications.

We were now getting lots of quality data flowing into the warehouse. The equipment logs were a good source of data as we were able to start to show trends where before there was only alerts on failure. Door access showed trends where office buildings weren’t being used as much as they had thought, and the environmental system reports revealed rooms were being heated with no one in them.

To show value early, reports were built around the separate concerns. This raised interest in the solution and we began linking the different data sets. Linking equipment performance to temperature and humidity reading from the environmental system for example. Room temperature against office occupancy, and others.

The value showed by the reports encouraged and expansion of the sensors. Electricity and gas monitoring, solar panel efficiency, individual room occupancy, and so on. One interesting metric was to monitor the room temperatures against external temperature. From this it was possible to calculate the heating or cooling efficiency of individual rooms, and it provided evidence where either heating or cooling was failing or not performing as needed.

Normally separate data sets are now combined to reveal insights previously unseen.

Buildings can be measured for efficiency and occupancy. Metrics can be created to show when intervention is needed.

Observing trends can see off potential problems before they become a risk. 

All the data needed to make important decisions is all in one place and accessible.