Originally written by Matt Lawson, MDT Director of Technical Operations for the SE Blog
Is your plant protected by against human error, equipment failure, etc.? Who is making changes to the automation devices in your plant? Where is the latest version of the device program? What changes have been made to your device programs?
If any of these questions cannot be easily answered, your plant is at risk for downtime, product loss, safety hazards, security breaches, loss of intellectual property and more. The increased use of plant floor automation to achieve production goals has created a dependency on industrial programmable devices such as PLC, CNC, HMI, Robots and drives. Many of these programs are changed as adjustments to variables and logic are needed to continue smooth operation, but these changes also put the plant at risk. What happens if someone makes a change to a program that results in undesired performance, or corrupts the program due to inadvertent changes? If there is no backup the result is costly downtime.
For example, take one particular large soft drink manufacturing plant, producing up to 4,000 cans per minute and 3,280 bottles of soft drinks per minute, that followed a manual, human driven change management process. The procedure included using two separate back-up systems: a ‘master’ and a ‘working’ system based on manually copied files. Technicians changed PLC code locally and then followed a set of procedures to ensure the change was recorded and backed-up, making the plant very vulnerable to human error. This plant unfortunately discovered that if a change was unauthorized or the engineer simply forgot to save the change to the main back-up file, version control issues resulted. If just one production line went down because a PLC has fallen over due to poor software version control, the plant was faced with downtime costs of around $1,800 per minute.
To avoid future costly errors related to version control, this plant installed a Change Management System (CMS) to maintain an ancestor history of all PLC and non-PLC files on site. Particularly useful to the plant is the CMS’ schedule compare feature that automatically compares the program in the device with a stored program. This feature detects and identifies changes between the programs so that designated users are immediately notified of changes that may have been unknown or unauthorized.
Shortly after installing the CMS, the plant had a problem with a PLC controlling the water treatment plant resulting in all the PLC’s code to be lost, stopping all water supplies to the production process. Production was up and running within five minutes by downloading the correct code from the CMS program repository. On another occasion, the main preparation PLC fell over. The code was reloaded immediately over the network from the CMS and the plant incurred no downtime. In both cases, the reloaded code was up-to-date due to the schedule compare capability which updates the ancestor files if discrepancies are found during the compare routines. Few discrepancies are now detected, as the on-site technicians now use the CMS for checking out programs and verifying them again when making a new revision.
In a world where mistakes and unexpected failures never happen, an Automation CMS would not be necessary. Since that world does not exist, having a CMS is an imperative.