Pipeline Performance Measures Feedback Summary Report – What We Heard

Pipeline Performance Measures Feedback Summary Report – What We Heard [PDF 767 KB]

Copyright/Permission to Reproduce

July 2023

Document Summary

This document is a summary of the feedback on the proposed update to Pipeline Performance Measures (PPM) which was provided by the specified PPM companiesFootnote 1 regulated by the Canada Energy Regulator (CER).

The specified PPM companies provided feedback on the proposed update and provided suggestions for additional improvements. Though the companies generally agreed that most of the proposed updates would improve the data collected, some overarching concerns were identified on the variation in management systems within the industry and the low applicability of the current and proposed PPM for regulated companies.

We heard that most companies do not use the PPM or the aggregated data report for benchmarking their own management systems. Instead, they rely on internally developed measures and/or other industry measures such as PHMSA/CEPA/EPRGFootnote 2, and as such the current and proposed PPM are not considered useful for CER regulated companies that provided feedback.

The specified PPM companies stated that they would find more value in issues and trends reported by the CER from its continual oversight of regulated companies (for example, through incident reporting, compliance activity, and audit findings) rather than continuing with the PPM.

Table of Contents

1. Background

In 2012, the National Energy Board (NEB), now known as the CER, decided to introduce PPM reporting requirements to promote the continual improvement in the management of pipelines, assist in compliance verification planning and to determine whether companies were providing information consistent with its existing knowledge. The NEB began collecting PPM in 2013 from a select group of regulated companiesFootnote 3. The PPM data submitted is aggregated and published annually as a report on the CER website, allowing regulated companies to benchmark their pipeline performance and improve their management systems.

In 2021, the CER developed and proposed new measures using the initial goals developed for the PPM project to enhance its regulatory oversight. On 18 November 2021, the CER issued a notice to specified CER regulated companies soliciting feedback on the updated PPM by 19 January 2022. The CER directed companies to focus the feedback on the feasibility of collecting and reporting the proposed measures along with the value as industry benchmarks. This document contains a summary of the information we received from the companies.

Following the submission of the written feedback, the CER met with four regulated companies that had requested additional virtual follow-up meetings to discuss proposed changes and seek clarification. Based on this consultation and an analysis of the data received over the past 11 years, the CER determined that the PPM program has served its purpose of guiding companies in developing their management systems and is no longer necessary. As a result, the CER has decided to close the PPM program and will no longer require companies to submit PPM data. Instead, the CER will use other methods to ensure continued oversight of regulated company performance measures that are mandated by the Onshore Pipeline Regulations.

Top of Page

2. Virtual Meeting Questions and Feedback

The CER identified four key questions for discussion in the follow-up virtual meetings with companies:

  1. Are the current annual data summary reports useful for benchmarking?
  2. Is there any better way to report PPM data back to PPM companies?
  3. Would annual meetings to review PPM trends and observations be useful?
  4. What can we collect that informs performance against the OPR 6.5(1)(e) requirement (Risk Assessments)?

2.1 Are The Current Annual Data Summary Reports Useful for Benchmarking?

We heard that most companies that report PPM do not use the aggregated data for benchmarking their own internal performance. Some companies indicated that they do review the report and key measures for insights into industry performance in general.

Rather than using the CER published PPM data, we heard that companies tend to rely on their own internally developed measures as well as other industry associations measures for benchmarking purposes. Larger Group 1Footnote 4 companies have assets throughout North America and we heard that they tend to use PHMSA/CEPA/EPRG performance measures as the data is more comprehensive and granular than what the CER publishes. The PHMSA/CEPA/EPRG data also contains information and key factors on pipelines that are outside of Canada that companies said they find useful when comparing peer-to-peer metrics and industry averages. CEPA measures are also based on common Canadian Standard Association (CSA) framework which companies said gives consistent information across industry. Some companies indicated they do not use PPM to benchmark internally because PPM are not aligned with industry standards like CSA Z260 Pipeline System Safety Metrics that companies now rely on for loss of containment measures.

Companies observed that only a small portion of the current measures are applicable as management systems differ between regulated companies. We heard that there have been attempts for industry to use select information from the published PPM data in the past, the information was not useful. For Emergency Management measures, companies said that the data was initially unique at the time, but the count of activities was an impractical metric, and couldn’t be interpreted as good or bad.

Companies suggested that by providing the required PPM, the CER is somewhat indicating that they are the only measures that companies should be collecting. Several companies also noted that the collection of data for the PPM submission required significant effort for verification and internal review before submission.

2.2 Is There Any Better Way to Report PPM Data Back to PPM Companies?

When asked whether PPM data could be presented back to companies in a better form than the annual PPM Data Summary Reports, there was a mixed response. Some responders agreed that individual company meetings to review trends that the CER was observing in the data would be valuable and others indicated that as long as PPM data are in their current form and aggregated as they are for publishing they would not be useful. As the measures currently stand, companies indicated that PPM are not considered useful for industry as PPM data is aggregated and does not provide any additional information to companies beyond the data that their current internal measures and industry association measures contain.

To increase the value of the proposed updates, companies suggested the following:

  • Align with the metrics contained in the existing CSA Z260 Pipeline System Safety Metrics (CSA Z260) and API RP 754 Process Safety Performance Indicators for the Refining and Petrochemical Industries (API RP 754)
  • Report measures on ‘type’ rather than a running total for all reporting companies. Breaking down the data into subcategories or a rate (per km) would be more beneficial for comparison rather than the current raw numbers
  • Improve the narrative on the results. While companies recognize and support CER efforts taken to maintain company confidentiality and minimize the public’s misinterpretation of results, additional messaging to the public on industry averages and how to interpret the data would also be beneficial.
  • Normalize the data based on the number of kilometers of pipe, or other common factors, prior to publishing the next report.
  • Engaging with companies when outliers within the submitted PPM data are identified. Companies could then provide some context around the numbers that are submitted.

It was suggested that the CER re-focus PPM to reporting issues discovered through its oversight such as through its audit process or reportable incidents. Companies were of the opinion that there is opportunity for more timely publishing of lagging information already collected by the CER along with industry trends for incident causation and contributing factors collected through its incident investigation. Companies said that the CER should consider providing performance quartiles and trending including guidance on measures with greater definition and examples of calculations.

Companies indicated that more insight into incidents would be helpful if they could break down incidents by program or component failure; by understanding the immediate and root causes of incidents, companies could obtain greater understanding of the failure mechanism (for example, knowing that a certain percentage of facility leaks had root causes related to management of change, how many integrity incidents other operators have had, how many near misses, what types of near misses have happened, etc.) and publish the data via a standards format (such as CSA Z260 Tier 3 and Tier 4). We heard that the ‘year over year’ measures aren’t as useful as constantly updated measures based on CER findings.

2.3 Would Annual Meetings to Review PPM Trends and Observations Be Useful?

We heard that with the current and proposed data submission requirements, and the inability to compare it to other equivalent operators, companies didn’t find much value in annual meetings. Companies indicated that if there were significant changes to the PPM, an online meeting could be beneficial in generating valuable discussions on reported measures for both the CER and industry, however, it would be a challenge to keep the reported data anonymous in this instance.

2.4 What Can We Collect That Informs Performance Against the OPR 6.5(1)(E) Requirement (Risk Assessments)?

Companies repeatedly stated that there are issues in narrowing risk assessment measures and the proposed changes down to a few Key Performance Indicators (KPI) as the process of determining risk is very complex and difficult to analyze (for example, issues are never related to just one threat).

Companies have concerns that requiring Industry to choose between quantitative risk assessment and qualitative risk assessment would result in parts of Industry choosing to report quantifiable likelihood of failure (LOF), consequence and risk results, compared to others who may use a qualitative or semi-quantitative risk assessment and that this could cause confusion in interpretation and published results.

Several companies responded that they are concerned about how submissions will be aggregated into low, medium, and high-risk categories accurately and effectively without including context such as if the quantitative estimates are fully quantitative or if model error is considered. Companies also said that they are concerned about submitting detailed QRA results simply to be input into three broad categories. Companies requested that the CER consider providing definitions of low, medium, and high for alignment of their risk results.

We heard that there are differences in capabilities and the maturity of the programs between companies that makes the comparison risk assessment data almost impossible. To compound on that, companies said that they add capability to their risk model continuously which could deviate further from the CER’s standard.

We heard that when it comes to risk assessment, context is important and is very challenging to convey through individual measures. Companies said that collecting the data in such a simplified manner misses this context and calculated values can be very subjective and that Without the whole picture, the data could be misleading and unhelpful for benchmarking purposes. Companies said that information that informs risk assessments also changes very rapidly depending on maintenance programs.

Companies suggested using CSA Z662 Annex B for risk assessment to compare qualitative and quantitative risk with the new changes. Another suggested approach was that the CER collect information on actions related to risk rather than the calculations; for example, “number of digs” is not a good indicator because of subjectivity, and no action after a dig might indicate that a company doesn’t have an escalation process in place. It was said that a better indicator might be “the number of actions escalated”, or “how many risks were actioned and mitigated”.

We heard that confidentiality is a concern with the updated risk measures, with some companies indicating that they would report a minimum amount of information to satisfy the request rather than supplying any confidential information.

Some companies responded that they have concerns that the PPM data creates dual data collection and assessment – one to meet industry metrics and/or company specific metrics and internal processes, and one to meet CER PPM reporting requirements. We heard that that this duplication of work and overlapping measures could lead to potential inconsistencies. We heard that companies would be required to expend significant effort to revise their collection and compilation processes which currently align with other internal Company processes.

Top of Page

3. Written Feedback Proposed Measures.

3.1 Safety Management Measures

Companies agreed with the changes to the performance measures and agreed that they represent improvements and address some of the feedback previously provided by the industry.

Most companies suggested slight modifications to the guidance to clarify information and provide more precise definitions. They said that this would allow more accurate data to be submitted from all participating companies and might allow some benchmarking from industry. Companies suggested normalizing the data based on the number of kilometers of pipe or other common factors.

Companies recommended removing the Corrective and Preventative Actions Measures as data collected will not be an indicator of performance. An example was provided that it is unclear if a company with no safety actions identified is a company which excels at safety or a company with poor inspections. Companies said that to be effective, this metric must have a corresponding service standard to ensure quality of inspection findings which is not available industry wide. Further, companies said that the measure does not differentiate between system improvements and case specific improvements.

3.2 Emergency Management Measures

Companies suggested that modifying the Emergency Response Exercise Measure and the Communication Measures by normalizing the data using number of operating regions or asset length would be beneficial and might assist in benchmarking their own management systems.

Companies also suggested removing the Training and Competency Measure due to ambiguity as “employees directly involved in operating of the pipeline” is not the same as employees who could be called on to respond to an emergency, as well as the Coordinating Operational Activities Measure as it is not valuable for benchmarking. Companies suggested that the metric will vary due to company size and there is no correct target.

3.3 Integrity Management Measures

When referring to the proposed Integrity management measures most companies suggested modifications to the guidance to clarify information and provide more precise definitions, saying that this would allow more accurate data to be submitted from all participating companies and might allow some benchmarking from industry.

We heard that there are concerns with the updated measures as expertise and maturity of risk assessment methodologies can vary greatly between companies and the results could lead to inconsistencies in reporting performance measure results.

Some companies agreed with proposed changes under this section as the data could be provided within their current risk models. However, there was concern that with the different methods used throughout industry it would be hard to obtain accurate comparable results for benchmarking once all data was aggregated. Furthermore, it was suggested that the aggregated data published on the CER website may be taken out of context and misunderstood by the public.

It was suggested that to promote consistency in reporting across industry, definitions should be established for what the CER considers Low, Medium, and High risk. Most of the larger Group 1 companies said that they currently perform more granular risk assessments and that common definitions would be helpful in promoting consistency in reporting this metric. It was suggested that the addition of meta-data may provide more context as to the Risk Analysis methodologies.

Concerns were also raised about the guidance for Integrity Management Performance

Measures which allows companies to choose between a quantitative risk assessment (QRA) or qualitative risk assessment. Companies said that this could result in parts of Industry choosing to report quantifiable LOF, consequence and risk assessment data compared to others who may use a qualitative or semi-quantitative risk assessment, and that this would create inconsistency for companies utilizing QRAs captured in the broader categories of low, medium, or high risk without context.

In the long term, some companies recommended waiting for the 2023 version of CSA Z662 to be released allowing Industry to adopt the informative annex as a means of driving more consistency in the way that risk assessments are performed.

Companies also raised the concern that the updated measures require them to report information that is confidential and critical to internal planning processes required to manage hazards and risks.

3.4 Environmental Protection Measures

In general, companies did not oppose the adoption of the proposed Environmental Protection measures and believe that it is an improvement from the previously collected measures. There was some concern from some companies that they may not actively collect environmental issues data at this level of detail.

Most companies suggested modifications to the guidance to clarify information and provide more precise definitions to allow more accurate data to be submitted from all participating companies and allow permit some benchmarking from industry. Companies suggested including a list of examples of Operational Environmental Issues aligned to the sub-categories.

3.5 Damage Prevention Measures

No significant comments were provided for the updated Damage Prevention Measures.

3.6 Security Measures

No significant comments were provided for the updated Security Measures.

Top of Page

4. Conclusion

Since 2012 when the PPM measures were introduced, PPM data has been submitted annually, aggregated, and posted on the CER website in a report allowing regulated companies to benchmark their pipeline performance against industry averages and inform improvements to their management systems and performance.

Through the consultations that were held in 2022, the CER heard that companies are no longer using the PPM or the aggregated data report to guide the improvement of their management systems, or for benchmarking purposes as originally intended. As company management systems have matured, they now have and maintain internal company measures that are inherently more effective because they were selected intentionally by companies to relate to their individual goals, objectives, and targets specific to that company’s management system. The PPM program has helped companies develop and improve their management systems.

Over the past 11 years, we found we were able to gather the safety information we need from existing processes like in-depth audits, inspections and the detailed data companies must report when an incident occurs. The PPM project has served its purpose in guiding companies to develop their management systems and it is no longer necessary.

After the consultations with our regulated companies and consultations with the subject matter experts within our own organization, we have decided to close the PPM program. The CER will no longer require companies to submit this data. The CER will explore other ways of targeting oversight of regulated company performance measures as required by the CER’s Onshore Pipeline Regulations.

We are a learning organization. When we see there is something we can do to improve how we work, we act. By removing these reporting requirements, it reduces the regulatory burden on companies and supports overall competitiveness.

We will continue to monitor that companies are following the rules of our Onshore Pipeline Regulations and have effective management systems to ensure pipelines are safe.

Top of Page
Date modified: