When OEE looks good but isn’t accurate

Why robust shop floor KPIs don’t appear on the dashboard

In mass production today, stable operations are almost inconceivable without a high-performance IT infrastructure. Variant management, material flow, quality assurance, maintenance, traceability and production metrics are all interlinked. MOM (Manufacturing Operations Management) systems play a central role here: they connect ERP, the shop floor, plant, processes and reporting.

This role is particularly demanding in automotive or life sciences environments. In these sectors, the focus is not only on efficiency, but also on quality, traceability, compliance and operational stability. At the same time, the reality on the shop floor is rarely straightforward. Many production environments are brownfield sites: different types of plant, various PLC generations, multiple manufacturers, legacy interfaces, old and new data models, and bespoke process logic all coexist.

This does not make MOM any less important. However, it does mean that these systems must be continuously reviewed and further developed from a technical perspective.

Key performance indicators now play a vital role in managing modern production. Andon boards are used as a visual management tool in lean management to ensure real-time communication during ongoing operations and to highlight problems as well as the current status of production. Figure 1 shows an example of an Andon board.

The data for key performance indicators is calculated within Manufacturing Operations Management solutions. Aggregated key performance indicators are derived from the operational data for tactical assessment by managers responsible for production.

Suppose that, during the morning meeting between production management, logistics management, quality management, maintenance management and shift supervisors, the OEE appears stable, machine utilisation is within the target range and the quality metrics seem plausible; yet the experience of those involved on the shop floor suggests that something is not quite right.

It is precisely here that the reliability of the key performance indicators and the quality of the data are determined. This is crucial in determining whether MOM merely provides figures or is used as a genuine basis for decision-making on corrective actions.

Many are familiar with this scenario: the key figure looks good, but operational problems are being reported. The product changeover took longer than expected. The plant is not as stable as the OEE suggests. Rework is known to be occurring, but the non-conforming output (NIO) figure and cycle time look good. Furthermore, the classified downtime figures do not reflect the real problems.

At first glance, this is a data problem. On closer inspection, it is an organisational problem. This is because key performance indicators such as OEE are only reliable if the technical reality of the shop floor is accurately reflected in data models, processes, status logic, counting mechanisms and validation rules.

This is precisely where a key weakness of many digitalisation initiatives becomes apparent: modern MOM systems require ever more data. But more data does not automatically mean more truth.

A dashboard can only display what has previously been correctly recorded, interpreted and calculated. If the underlying logic does not match actual production, a dangerous situation arises: the organisation trusts figures that look plausible but are not robust.

Take OEE, for example. On paper, it is a well-known performance indicator. In practice, however, its significance depends on many details. Figure 2 shows the data flow that feeds into the OEE calculation. There are many pitfalls here:

When is a piece of equipment considered available? How are micro-stops assessed? Which statuses are received from the PLC? Which downtimes are planned, and which are unplanned? Are faults classified correctly? Are performance losses accurately identified? What if rework is known to be required? What if inspection plans do not match the current product variant? What if OK/NOK decisions are technically processed but not correctly interpreted from a business perspective?

In such cases, no technical error occurs in the traditional sense: The system is running, the interface is transmitting and the report is generated – but the business-related information is incorrect.

This is precisely what makes KPI quality so challenging. It is not enough simply to collect data. Nor is it enough to calculate a key performance indicator technically. The crucial factor is whether the data logic accurately reflects the actual behaviour of production. And this question cannot be answered by IT alone.

The underestimated success factor here: key users.

This is where the role of key users comes into play. Key users are neither advanced users nor ‘power users’ in the traditional sense. In a functioning MOM environment, they act as business interpreters, plausibility checkers, prioritisers, testers and multipliers. They are familiar with shop floor processes, bottlenecks, typical disruptions and the unwritten rules of a production line – and know how these are configured in the MOM system.

Key users know when a key performance indicator (KPI) merely looks good on paper. They recognise when a new screen is technically correct but does not work on the shop floor. They notice when a release alters data logic that will later have a direct impact on OEE and other KPIs.

MOM is the digital backbone, but not an autopilot

Key users are therefore a crucial link between production and IT. They provide the context that is often missing from tickets, interface specifications and data models.

Key users do not just ask: ‘Does the application work?’ They ask: ‘Does the application accurately reflect our process?’ And, more importantly: ‘Can we really make decisions based on this data?’

Without subject-matter validation, digitalisation remains risky.

Many companies invest heavily in systems, platforms and dashboards. This is both right and necessary. However, the effectiveness of these investments is not determined by technical architecture alone. It depends on whether the shop floor uses the systems, whether the data is accurate, whether releases are validated from a business perspective, and whether deviations are detected early on.

Particularly where there is a high frequency of changes due to continuous improvement processes, new product variants, process adjustments, regulatory requirements or security issues, these tasks become ongoing. MOM is not a one-off project, but a continuous operational and improvement process.

That is why roles are needed to bring subject-matter stability to these changes. Key users fulfil precisely this role. They identify pain points on the shop floor, formulate business requirements, assess the impact on key performance indicators, support testing, and assist with training, rollouts and the collection of feedback following a release. Or, to put it another way: they ensure that digitalisation does not proceed without taking the shop floor into account.

Figure 4 depicts key users as anchors of reality. They define the business requirements that result in rules for process and data quality. These rules must be applied automatically to the raw data during real-time operations. Deviations must be detected, and data and process quality must be ensured either through corrections in the data stream or through timely patching.

In addition to static rules, AI agents can also provide automated support for this task and act as virtual key users.

Ultimately, it is not just about OEE. It is about trust and insights into the causes of waste: the trust of workers in the systems, the trust of managers in the key performance indicators, the trust of IT in technically sound requirements, and the organisation’s trust that digital decisions are based on a robust foundation.

If the OEE looks good but is incorrect, this is more than just a reporting issue. It is a warning sign. It indicates that a gap has emerged between the system and reality.

Key users are the ones who can recognise, identify and close this gap. That is why companies should not view key users as playing a secondary role. They are one of the most important success factors for data-driven production.

After all, reliable KPIs do not emerge from a dashboard. They arise where system logic, process knowledge and shop floor experience come together. And that is precisely where good key users operate.

adesso does not merely provide support for the implementation of IT projects. We offer comprehensive support – from value stream mapping and potential analysis right through to reliable implementation and training for stakeholders on the shop floor.