Alarms and Operator Intervention
(A Flawed Safety Layer)
Most companies rely on operators to detect, diagnose, and intervene during abnormal situations. Plant managers and safety personnel consider operators to be a critical safety layer. However, we cannot expect operators to do a very good job if we do not focus on human factors, addressing situation awareness, ergonomics, to improve the way we operate our control rooms. [1]
How often does a control loop fail, a pump spring a leak, or an operator make an error? Everything fails, it is just a matter of when. Multiple, independent safety layers are designed with the intent that if something fails, other layers will prevent the failure from escalating into a hazardous event.[2] It is important to have multiple layers, but that does not mean we can overlook the importance and strength of each independent layer.
The “Alarms and Operator Action” safety layer may be the most overlooked and neglected safety layer in the industry. Sure, many companies have spent a lot of time and resources on alarm management, but what about the human factor, the operator. Rationalized alarms do not ensure proper operator intervention. Just think about the operator working in a dark control room, on his 16th straight hour, staring at one of 10 flashing graphics, with a full alarm summary screen, reviewing a poorly written procedure as the supervisor is leaning over his shoulder.
Each safety layer can fail, that is why we have multiple safety layers in place. If the basic process control system fails, and the operator fails to respond in time, the SIS will likely prevent any additional damage. Is that good enough? No, we need to think about operators as much more than a safety layer because they have a significant impact on the entire business.
James Reason makes it very clear in the following illustration that each layer of protection often contains weaknesses. In the illustration, each slice of Swiss cheese symbolizes a layer of protection and each hole represents a weakness. The assumption is that major catastrophic events occur because weaknesses in each safety layer eventually line up and create failure. Knowing the importance of the operators job should motivate us to address every potential weakness in this particular layer.
Alarms are configured to alert operators that they need to respond and manage an abnormal situation. Many managers have focused on reducing alarms to achieve the EEMUA 191 KPI’s, following the ISA 18.2, and IEC 62682 standard. However, they should be focusing on the operator’s ability to maintain the process in a safe, economical steady state. To achieve this, we must look at “time in alarm,” to reduce the costs associated with running the process outside the operating limits; we call this “abnormal operations”. How long you are operating or running a process in “abnormal” can have an enormous impact on equipment; quality, and safety.
When alarm setpoints are configured perfectly, the alarm allows operators enough time to diagnose the problem and manage the situation before they reach what we call “abnormal operations.” This is the point where they have exceeded the amount of time allocated to detect, diagnose, and manage the situation, and where they are dangerously close to the consequence of the alarm. Once in “abnormal operations” they are now operating past the safe, economical, or qualitative threshold, the very situation that operators are hired to prevent.
See below an image depicting operator response to abnormal situation, from the IEC 62682:
IEC FDIS 62682 @ IEC 2015 Figure 5 – Feedback model of operator-process interaction
It’s really tough to put a dollar sign on each second you are operating in “abnormal,” in some cases it is a safety issue with economic consequences, and in other situations operating in “abnormal” can be a quality or regulatory issue. You may just be putting additional stress on equipment and jacking up maintenance costs. Studies have determined that operating in “abnormal” costs our industry billions of dollars per year. The point is, you set alarms so operators can keep out of “abnormal.” If you knew that each second in “abnormal” cost you $1,000, would that make a difference? Why? You set the operating limits for a reason; your goal is to operate within those limits and prevent the economic, safety, or environmental consequence from happening.
Something to think about: Unplanned downtime is costly; can operators prevent downtime?
Most alarms are not set to give the operator enough time to detect, act, and resolve problems. In many cases alarms are not prioritized correctly either, an operator might be working on a simple problem while a much more urgent situation is growing out of control. Alarms should be prioritized based on the safety impact, financial impact, and environmental impact as well as how much time an operator is given to manage the situation.
During customer projects, we often see alarm set-points that have been adjusted over time based on operator feedback. We also find that most alarm set-points are not accurate. Each of the following elements: time to detect and read the alarm, time to act; time for the process to react to the operator action, and the time needed for the process to calm down have not been assessed for each alarm. Reference the image below from the IEC 62682 alarm management standard:
IEC FDIS 62682 @ IEC 2015 Figure 4 – Alarm response timeline, modified by Ian Nimmo & David Lee
Question: How well are your operators responding to alarms? To measure this, we don’t focus on how long it takes the operator to acknowledge the alarm. We want to know how long it takes operators to act, to make a real change in the process and how long it takes the process to recover after the operator acted. To determine the effectiveness of the operator response time it is imperative that you measure and evaluate the time between the alarm activation and when the operator makes the change (Operator takes appropriate action). This will determine the “operator response time”. You should also measure the time from the operator change to when the alarm is cleared, this will determine the “process delay time”. If you do not measure this, you will not know if you have issues with alarm set points, priorities, and or if you have issues with operator response.
Let’s look at an example of an incident that happens all too frequently: A tank level alarm activates, the tank is too full, it is rising, then the temperature alarm activates, the chemical inside the tank is too hot. The operator has 5 minutes to act. The operator detects the problem and is responsive. The operator receives and acknowledges the alarms within 30 seconds. The operator acts and reduces the flow into the tank, starts the cooling process, and opens a relief valve. This takes a minute. The control system receives the changes and begins to activate the commands. This takes 45 seconds. The problem is being managed and returns to a normal state 3 minutes later. Total time in alarm: 4.75 minutes. All ends well in this situation because in this example the process never went outside the operating limits. It almost did, but it never reached the “abnormal operations” threshold. This alarm is configured correctly, and the operator managed the situation in time.
What if the operator was distracted and took a minute to work on the alarm? The process would have been in “abnormal” for almost a minute. See the image above. What damage can happen to the tank for the excessive heat and high level of the fluid? Could the quality of the fluid be jeopardized? What could have happened if the operator made a mistake and did not manage the situation properly? In many cases, operators are focused on other tasks and may have a high workload so they can be slow to detect and act during abnormal situations.
What could have happened: Overflow of hazardous liquid (environmental impact) – clean up and fine (financial impact) if someone was in the area and was affected by chemical contamination (safety impact). The above examples occur more often than we care to imagine. What we need to do is change the culture. The operator should have clearly known the tank level and temperature was rising way before the alarm occurred.
The lack of situation awareness is a major issue in our industry. Most operators rely on alarms and do not have a clear graphical interpretation of what is happening in the process. We need to address the demons of situation awareness per Mica R. Endsley’s book, Designing for Situation Awareness: An Approach to User Centered Design:
· Overloading of information including HMI’s and Alarms
· Salience Issues with Displays
· Short-term memory issues
· Out of loop syndrome, like sitting with your back to control system while working on IT PC.
· Attention tunneling
· Errant Mental Model
· WAFOS (Workload, Anxiety, Fatigue and Other Stressors). [3]
Based on Ian Nimmo’s slide describing Abnormal Situations
Looking at the image above, you will see three process states: Number 3: Normal operations, Number 2: Abnormal Operations, and Number 1: Abnormal operations for an extended period.
Focus on number 2 – An abnormal situation takes place, the alarm activates, the operator detects the alarm, identifies the problem and resolves the issue relatively quickly, thus minimizing the amount of time that the process is in “abnormal.” This put a strain on the equipment and had an adverse impact on quality, but it was detected and managed by the operator relatively quickly. Is this acceptable or do we need to improve reaction time? Have we set the operating limits correctly or are we operating outside the operating limits on a regular basis?
Now look at number 1. The alarm activates but the operator has a high workload, so this abnormal situation goes unmanaged for a long time. The issue creeps up closer and closer to an emergency situation. Finally, it is detected and managed just shy of a costly emergency shut down. This put a lot more stress on the equipment, had a significant effect on quality and could have escalated into a serious incident where lives were jeopardized.
A major reason why operators continue to operate outside the operating limits is called the “Key Hole Effect.” Your control room operator is monitoring a process with several loops and several points. They have multiple screens, and each screen has hundreds of graphics to navigate through, uh oh, several alarms start to activate! The operator begins to navigate through the graphics to determine the problem, but has a very limited view because there are so many screens to click through. We are operating outside the operating limits, and this is about to get ugly.[4] The operator is on the tail end of a 12-hour shift, ten screens are flashing all at once, he is trying to figure out what is happening, he is struggling. The supervisor is out in the field. The radios are down, and the clock is ticking!
Based on a slide from Mark Green of HCD in Norway
The image above is a real example of the number of screens that were selected after a few alarms activated. The operator navigated for several minutes in an attempt to resolve the problem. During this time, the process was outside the desired operating limits[5].
Data Overload or the Key Hole Effect causes a delay in response time and contributes to running the process in abnormal, putting a strain on equipment and contributing to higher maintenance costs.
Most operators are reactive versus proactive. Why? Because they have a ton of information to process. They have several graphics cluttered with data and numbers that require mental calculations to process the information. Most operators do not have a good way to identify where the problems originate and what to do, so they are forced to rely on alarms. Hey, you might get lucky and have an operator on the right screen at the right time, but it does not mean he is going to see that increase in temperature amongst all the other data on the screen.
Ideally we want operators to detect problems or see them coming and prevent the alarms from activating. To accomplish this, you can use a large situation awareness screen (a large overview graphic of the operator’s process) to combat “change blindness” and to draw attention to important information so the operator is aware of what is happening and what might happen.
As an industry focused on safety, we have an opportunity to change the culture. The idea is to encourage operators to maintain the process in a steady state, predict abnormal situations, and act accordingly. If operators wait for alarms before they act it will extend the amount of time they are operating outside the operating boundaries.
Why does it matter? If we are operating outside the operating limits the process is unstable and with each minute the situation escalates. Alarms are set to protect assets, equipment, people, quality, and reduce waste. If an operator gets 100 alarms during a shift and each alarm is in "abnormal” an average of 3 minutes each, that means that you have been operating outside the operating limits for five hours. Is that okay? It shouldn't be, that's why we have alarms and operators, they go hand in hand as a safety layer.
We often worry about how many alarms we get during a shift but we need to focus on the amount of time that we are in "abnormal". An alarm is a problem that requires action! We have operators in place to intervene and return things back to a steady state, they do this through observation, procedures, experience, training, and communication. Unfortunately, we keep seeing issues with the operator’s environment, user interfaces, and management systems. These issues prolong "time in alarm" and make it difficult for operators to solve problems.
You may have put together a great alarm management program but what good is alarm management if your operator response time is flawed? Are you measuring this at your site and have you determined what the negative effects are?
Companies spend millions of dollars on asset management. One of the biggest missed opportunities is likely the biggest asset "the people". Companies spend millions of dollars staffing, training, and filling important positions like process operators but they don't spend the time to design the user interface to support the objective - to operate within the desired, safe, set boundaries of the operating limits.[6]
Many managers believe they have already spent a ton of money on graphic design. In most cases, the DCS vendor and engineers simply copied P&ID's and never spent money on user centered design, specifically human factors design. One of the biggest missed opportunities is when companies spend money and time on alarm management but do not invest the same on the graphics with a specific focus on situation awareness and human factors.
We’ve studied hundreds of control rooms and see multiple screens, poor overview displays, lots of color, no human factors philosophy document or style guide in place, multiple graphics, poor navigation, lots of data, the lack of trends, no analog objects, green and red everywhere, mostly just cluttered depictions of P&ID’s.
Many of our clients have focused on alarm management but have not invested in applying human factors to the design of the user interface. Point being, if you get the graphics right, you can significantly reduce alarms and stay out of “abnormal”. Remember, the idea is to have proactive operators that predict problems rather than reactive operators who rely on alarms.
The graphics need to be designed with a human factors expert that understands the process, an experienced console operator, and process engineer. This team of three is critical to the success of the design. We have to bring the big picture back; we have to use analog objects that reduce the need for mental calculation and trends that enhance situation awareness. The information needed on the screens must be based on a thorough task analysis. Until we get this right, operators will operate in abnormal, and companies will continue to lose money and jeopardize safety. Large overview displays are customized and individually designed for situation awareness screens, drawing attention to the highest priority issues that the operator should be aware of.
I talk to several engineers every day and hear the same things: “We are doing some things to address these issues, but we could do better.” “We do not have money in the budget for this, but it is a good idea.” “Management does not understand this.” Solution: Arrange an educational training at your site and invite upper management.
We have dedicated resources to optimize the process - now we need to optimize the operators by applying human factors and improving situation awareness.
Here is the most important aspect to grasp: How long we are operating outside the normal safe operating limits will have consequences. That is why we set alarms and have operators ready to act when abnormal situations occur. Operators are the pilots of the process, they must be able to detect, diagnose, and manage abnormal situations.
Have you measured your operator’s workload during normal and abnormal situations? Do you have the right number of operators, are the supervisors where they should be; and will your operators be ready when the unthinkable happens?
Ever since the introduction of computer-based systems, our industry has experienced organizational accidents, which have been catastrophic in nature. “Organizational accidents are a product of technological innovations that have radically altered the relationship between systems and their human elements.” Investigations have revealed the major issues and research has identified best practices to either prevent or mitigate the consequences of such accidents.[7]
Does your control room environment, staff, and operator management systems effectively control and operate under all conditions? You have to make sure you have the right number of operators and that the supervision and work team design is structured to support day-to-day operations as well as any abnormal situations that may occur.
An operator Staffing/Workload assessment can tell you where you are overloaded and how well they can respond when alarms activate. A proper staffing study is based on science, where each position is measured to determine how many operators you need for normal safe operations and how your staffing arrangements will be effected during abnormal situations.
A workload study will not only show you where you are overloaded and under-loaded but it will also identify breakdowns and gaps in situation awareness and human factors. Poor staffing arrangements, poor situation awareness, and high workloads contribute to human error, a major problem that has been overlooked. Operators have taken on more and more responsibility over the years, on top of that our veteran operators are retiring and new operators are struggling with out of date procedures and informal training.
The following chart shows you each operator position in blue, and what the workload score is for each position. Notice position number 1 has the highest workload. Surprisingly, this operator was the least experienced and the operator with the lowest workload was the most experienced.
An operator workload study allows a client to identify where job duties are unequal and should be reorganized, consolidated, or new duties assigned. It also identifies management systems that are not meeting best practices. Management systems are the systems that support the effectiveness of the operator like: Alarm Management, HMI Design, Training, Procedures, Shift Handover, Supervision, etc.
What about the control room design? The control room layout will have an enormous impact on situation awareness, communication, fatigue, and problem detection. If you have an opportunity to combine control rooms or build a new one, you should perform a staffing study, make sure you have the right number of operators first, then design the room to support their tasks and objectives. Common issues with the operator’s environment are:
• Operators sitting in the dark
• Poor environmental controls
• Poor communications and collaboration
• Distractions – phones, people walking through, staff seeking information, maintenance requiring permits
• Acoustics /Vibration
• Glare
• Console placement[8]
When we have high workloads, poor situation awareness, communication issues, poor ergonomics, alarm floods, hundreds of graphics, more than six screens to monitor, etc., we are more prone to make mistakes. Human Error has been identified as the leading cause of industrial accidents. Human factors must be assessed and addressed to reduce human error.
We do not purposely go out of our way to having an accident, but as the saying goes “to err is human.”
Dan Rather once said, “To err is human but to foul up requires a computer.” Izquotes.com
OSHA PSM inspections still report that 33% of PSM citations relate to procedural failures; a contribution that has not changed since the introduction of Process Safety Management. As an industry we are very good at finding technical solutions and fixing technical problems however, we are weak at addressing human factor issues and how people interact with technology, or how we behave. Some of the issues may still be technical, i.e. the accuracy of a procedure, but a bigger problem exists. That is getting operators to use the procedure by providing it in a useful and easy to follow format. Utilization and effectiveness of procedures are also impacted by the operating culture.
Training is another management system that is often flawed; it should not be so difficult to provide a good training program. Major issues include:
· Lack of competence
· No formal written or out of date training materials for console operators
· Flawed culture and use of procedures
· The performance of the training system has not been examined
· Small or low % of employees is completing: general safety awareness training, emergency response training/drills, technical training, etc.
· Supervision does not test staff competence
Even with good alarms and high-performance graphics, we see communication problems. Approximately 20% of incidents involve breakdowns in shift-to-shift communications. Breakdowns such as Piper Alpha will dominate our industry when situation awareness is compromised between shift workers. Other incident examples where communication issues contributed to the loss of life: Esso Longford and BP Texas City.
Supervision is also crucial. We often see one Supervisor trying to work between the field and a central control room. They often find themselves in the wrong place at the wrong time. They need to be where they can provide the most influence and direction.
In conclusion, operators need to be able to predict potential problems, prevent alarms, and reduce the amount of time we spend in alarm. We accomplish this by developing a situation awareness philosophy. We must be aware of the consequences when operating outside the proper limits. Operators have critical jobs. They rely on management to provide them with the tools, interfaces, and management systems that incorporate human factors so they can be effective at their jobs. Imagine running the facility without them, would you fly in a plane without a pilot? Not me!
Here are some of the major issues that contribute to “abnormal operations”:
· Situation awareness is compromised, so the alarm is ignored or missed
· Keyhole effect (lots of data on a small footprint) cluttered graphics
· Alarm floods bury a significant alarm
· Operators are complacent with working in abnormal and are not aware of the impact it is having on equipment and quality
· Operators may have a high or unbalanced workload
· Hands on training versus formal training
· Operating culture (complacent)
· Fatigue, health, stress, and excessive overtime
· Issues with the quality and accuracy of procedures
· Communication problems / breakdowns in shift handover
· Control room design / distractions / poor ergonomics, lighting, permitting
· Management believes “Alarm Management” alone should have solved this problem
“Operators are the pilots of the process – So we give them wings”
UCDS is a Control Room Human Factors Company with 100 years of combined expertise in control room operations. Our design services help operators detect, diagnose, and respond to abnormal situations so they can prevent major incidents, recover quickly, and operate within the desired, safe, set boundaries of the operating limits. www.mycontrolroom.com
[1] Session Fourteen: Layer of Protection Analysis and Common Mistakes
2013 Safety Control & Instrumented Systems Conference 1 Session Fourteen:
Layer of Protection Analysis and Common Mistakes Paul Gruhn, P.E., ISA Fellow
Global Process Safety Consultant, ICS Triplex | Rockwell Automation
[2] APA: Layer of Protection Analysis and Common Mistakes. (n.d.). Retrieved from http://www.idc-online.com/conferences/1304PaulPaper.pdf
[3] Designing for Situation Awareness by Mica Endslay, Betty Bolte, ande Debra G Jones page 31
[4] MLA: "Control Room Operators and the Key Hole Effect - User ..." Insert Name of Site in Italics. N.p., n.d. Web. 11 Oct. 2016 <http://mycontrolroom.com/blog/control-room-operators-key-hole-effect.html>.
[7] AFPM 2014 Annual Meeting March 23-25th Hyatt Regency Orlando, FL “The Human Side of Errors”
Ian Nimmo, Center Human Factors * Ergonomics LLC; David Lee, User Centered Design Services Inc. AM‐14‐08
[8] MLA: "Center for Human Factors & Ergonomics CHFE Managing the Risks." Insert Name of Site in Italics. N.p., n.d. Web. 11 Oct. 2016 http://www.prosys.com/content/general/documents/Managing_the_Risks_of_Organizati>.
See more of this Group/Topical: Global Congress on Process Safety