Over the years, working with data as an engineer in the aluminium industry, I encountered my share of problems. The impact of these problems ranged from missed deadlines to failed projects. Five common data problems stood out in particular.
1. Data silos
The different departments of any aluminium manufacturing plant have their own data systems. Even within the same department, there are many isolated systems. Isolated systems, instead of one single system, are desirable for process control: the production process becomes fault-tolerant. Unfortunately, analytics requires the exact opposite: different data from many sources must be combined into one large dataset for analysis.
A data system that is not connected to other systems is called a data silo. Examples of silos are the Manufacturing Execution System (MES), the data historian, and that large collection of Excel files stored on a network drive. The goal is not to eliminate data silos but rather to extract data from them and combine it in some other place for analytics. This data integration work is often vastly underestimated, especially if the data systems are closed, as discussed in the next section.
2. Closed Data Systems
Not all data systems play nice. Some systems deliberately make it hard to extract data from. It’s not uncommon for vendors to only allow exporting data in a proprietary data format that can only be read by the vendor’s programs. Some vendors offer expensive licenses that allow you to extract data in an open format (CSV, Parquet, JSON, etc.).
Closed data sources are a massive roadblock in digital transformation because they make system integration more difficult than it needs to be. This was one of the worst problems I encountered as a data engineer in the field. If your company is currently purchasing data systems, ask the vendor about the various data exporting options.
3. Faulty Backups
Focusing again on the individual data silos, I’m willing to bet you money that somewhere in your plant, there’s a hard drive that’s full while data is still being written to it. That data is lost forever.
I remember working on a large AI-project involving an aluminium cold rolling mill. Terabytes of process data were saved on a computer in the production facility. An automated daily script was responsible for copying these files over to the main network for permanent storage. Except – this script hadn’t run for the last two and a half years. After half a year, the hard drive was full. This means that data from two years was never saved. This project, which was estimated to save 300k USD next year, was scrapped the next day: there was no data.
4. Spreadsheet Storage
No other software tool has been so misused as Microsoft Excel. The fact that most office workers understand Excel is a double-edged sword: on the one hand, analysing data and sharing results is easy. On the other hand, whenever data needs to be captured, people default to Excel.
If your company is guilty of this, the result will look familiar: endless heaps of Excel files with similar data, in slightly different formats. After all, someone prefers comma’s, someone else uses dots. The files have slightly differing column names, or a different ordering of sheets. It’s incredibly hard to process data from spreadsheet files that aren’t formatted in exactly the same way. Often each file must be processed individually.
For better or worse, Excel is here to stay. The switching cost to a different tool is simply too high: people won’t bother. The solution in my experience is to use Excel for what it was intended for: ad hoc data analysis, making graphs, and sharing results. For long-term data storage, a proper database should be used instead.
5. Missing Data Context
Even if data is neatly stored in a database, this data could still be hard to interpret. The meaning of the values in a certain column can’t always be derived just from the column name. A process engineer may understand that TT_01_HF_PVC refers to the “temperature values from sensor 1 of holder furnace expressed in degrees Centigrade”. However, a data analyst, who’s not as familiar with the process, will not understand the data.
Contextualising data is a difficult task that goes beyond just properly naming columns and variables. Data contextualisation tools like data dictionaries have been developed to make interpreting data easier by providing descriptions that explain the data model to unfamiliar users. However, the disadvantage of data dictionaries is that they are essentially a parallel repository of information that has to be kept in sync with the actual database it documents. Some of this work can be automated, especially with the advent of AI-tools, but the devil is always in the details: an accurate contextualisation requires human ironing out the details.
Conclusion and next steps
If your company is serious about its digital transformation, take a moment to take an inventory of your data and data systems. Do you recognize any of the five common data problems? If so, address the problems in your data strategy. If you are not sure of a particular problem, feel free to reach out to me on my email denis@gontcharov.eu.
About Denis Gontcharov
Denis is a data consultant who helps aluminium manufacturers break down data silos. For the past five years, he has supported the aluminium industry with IT and data services as an independent contractor.
Previously, Denis was employed as a data engineer at Novelis in Germany, a leading aluminium rolling and recycling enterprise, where he played a pivotal role in transferring process data from production machinery to cloud systems. Prior to this, he was employed as a process engineer at TRIMET’s aluminium smelters in France and Germany, developing control software for the electrolysis process. Denis is a graduate in Materials Engineering from KU Leuven, Belgium, and is currently based in Berlin.
Decoding Data with Denis: In this exclusive AL Circle column, Denis delves into the evolving data management landscape within the aluminium industry. He explores how manufacturers are actively breaking data silos to integrate information across operations.
Keep an eye out for his column on the future prospects of unified data systems, highlighting their potential to enhance efficiency, decision-making, and innovation throughout the entire aluminium value chain.