Violation of FAC-008-3 R6 and FAC-009-1 R1 Results in $210,000 Penalty
Critical ManageEngine ADAudit Plus Vulnerability Allows Network Takeover, Mass Data Exfiltration
July 1, 2022Meeting Minutes Regarding the June 23, 2022, Reliability Subcommittee (RSC) Meeting have Been Posted on the MISO website
July 1, 2022
Summary:
Multi-Region Regional Entity (MRRE) group 15a includes Duke Energy Carolinas, LLC (DEC), Duke Energy Florida, LLC (DEF), Duke Energy Corporation (DECorp), and Duke Energy Progress, LLC (DEP) (collectively referred to as “Duke” or the “Duke Companies”). This violation consists of instances where DEC, DEF, and DECorp were noncompliant with FAC-009-1 R1 (extending to FAC-008-3 R6) for failing to have Facility Ratings that were consistent with their respective Facility Ratings Methodology (FRM). This violation involves instances of noncompliance that were self-reported, identified during a subsequent SERC-led audit, and during the extent of condition.
Additional Discussion:
Identification
SERC was first notified of this violation pursuant to an existing MRRE Agreement. The notification was a Self-Report on August 28, 2018, stating that the entity had discovered three instances where it had Facility Ratings that were not consistent with its Facility Ratings Methodology (FRM). All three instances occurred at a single 500 kV substation and each impacted a separate 500 kV line segment. The three line segments were incorrectly entered into the Facility Ratings Database (FRD) as bundled line conductors rather than single line conductors because at the time of the substation energization, there were no approved Facility Rating Spreadsheets (FRSs) available. In the absence of approved FRSs, personnel defaulted to the bundled line conductor ratings, which drove the conductor ratings higher. However, the conductors were the most Limiting Element and caused Facility Rating derates of 7.8%.
On February 8, 2019, while completing walk-downs in preparation for the SERC-led audit, DEC and DECorp discovered potential instances of noncompliance with FAC-008-3 R6, and submitted one Self-Report which was dismissed and consolidated with the original Self-Report. Additionally, during the SERC-led audit, SERC conducted sample walk-downs for all Duke Companies. While no issues were identified for DEP or DEF, the walk-downs identified that DEC had additional instances where Facility Ratings were not consistent with the respective FRM. The audit finding was dismissed and consolidated with the original Self-Report.
Following the SERC-led audit, SERC requested walk-downs from all Duke Companies to determine the extent of condition. The results of these walk-downs caused SERC to request that DECorp complete a walk-down of all Bulk Electric System (BES) transmission and generation Facilities, with an expected completion date in June 2022. DEC, DEF, and DEP later informed SERC that they would voluntarily complete 100% walk-downs of all their BES Transmission station and substation Facilities.
Walk-down Results
As of November 2021, the results of all walk-downs performed (prior, during, and post-audit) for each Duke Company are as follows:
- Of 326 DEC Facilities walked-down, DEC has a total of 65 instances where DEC failed to have Facility Ratings consistent with its FRM. A total of 22 of the 65 were subject to a Facility Ratings deration, resulting in a deration percentage of 6.7% for DEC.
- The 22 Facilities subject to derations had derates ranging from 4.39% to 82.82%.
- The derations occurred at Facilities between voltages of 100 kV to 230 kV. DEC’s issues consisted of instances where Elements were missing from its Facility Ratings documentation; instances where the wrong equipment was utilized in the ratings calculations; and instances where DEC did not correctly define its Facilities in its Facility Ratings documentation at one substation.
- Of 96 DEF Facilities walked-down, DEF has a total of 26 instances (including the three instances that were initially self-reported) where DEF failed to have Facility Ratings consistent with its FRM. Of those 26 instances, four Facilities were subject to a Facility Ratings deration, resulting in a deration percentage of 4.2%. Three of the four Facilities subject to Facility Rating derations had derates of 7.8% at a 500 kV bus. The fourth Facility was subject to a 31.7% derate on a 230 kV transmission line. DEF’s issues were generally related to discrepancies between field ratings and Facility Ratings documentation as a result of major equipment changes or asset commissioning.
- Of 725 DECorp Facilities walked-down, DECorp has a total of 69 instances where DECorp failed to have Facility Ratings consistent with its FRM.
- Of the 69 Facilities, 61 Facilities were subject to a Facility Rating deration, resulting in a derate percentage of 8.4%. DECorp’s Facility derations range from 2.3% to 35.1% in voltage ranges from 138 kV to 345 kV. The largest derate at a 345 kV Facility was 13.8%. DECorp’s issues center around a design practice in its FRM since the effective date of the Standard Requirement. The design practice did not allow for bus work or CTs to be the most Limiting Element. Thus, when a design mistake was made and the bus work and/or CT was in fact the most Limiting Element, none of the internal controls downstream could have identified the issues.
- DEP has walked-down a total of 69 Facilities and has not identified any instances where a Facility Rating was not consistent with its FRM.
Cause
The cause of this violation was rooted in the presence of vertical organizational silos, which created challenges to implementing Facility Ratings programs that were consistent and equally effective across the Duke Companies. The silos resulted primarily from the 2006 merger of Duke Energy and Cinergy in the Midwest footprint, now DECorp. Specifically, when the merger occurred, DECorp left a good portion of Cinergy’s Facility Ratings program in place, including an FRM that included the above-mentioned design practice. The combined FRM was different from the FRMs utilized by DEC, DEF, and DEP. In 2013, DECorp and DEC merged their FRMs and that design practice was removed from the newly combined FRM. However, DEF and DEP still had their own programs and FRMs. Around the same time, DECorp had already begun its efforts to correct the Facilities that were missing bus work and CTs. Those efforts were suspended prior to completion. Thus, although DECorp and DEC were using the same FRM, the two entities continued working in silos over the years. As a result of the silos, the Duke Companies had been working independently from one another, which resulted in a lack of uniformity and coordination over the years. This further contributed to a lack of awareness of the Facility Ratings program challenges that existed within the Duke Companies, including lack of or inadequate change management controls for equipment changes/replacements; inadequate controls in the asset commissioning process; and lack of documented procedures and controls to ensure that Equipment Ratings information and calculations used matched the corresponding ratings documentation.
Period of Violation
This violation began on June 18, 2007, when the Standard became enforceable and DECorp first failed to have Facility Ratings that were consistent with the FRM, and will end on June 27, 2022 once all walk-downs have been completed by Duke and all identified issues have been addressed.
While the Regions recognize and value Duke’s continuous commitment and resources expended in the creation and implementation of its overarching ICP, the issues that comprise this violation existed long before the ICP enhancements were implemented. Additionally, the Regions determined that the ICP continues to be hindered by the presence of vertical silos, which resulted in a lack of Facility Ratings program uniformity and coordination across the Duke Companies. As such, the Regions considered Duke’s ICP to be a neutral factor in the penalty determination.
Effective oversight of the reliability of the BPS depends on robust and timely self-reporting by Registered Entities. While Duke submitted two Self-Reports reporting some of the instances that comprise this violation, the majority of the instances were identified as a result of the field walk-downs performed, at the request of the Regions, to determine the extent of condition. Notably, the instances identified in the initial Self-Report were identified during a routine internal compliance verification. As such, the Regions are awarding Duke some mitigating credit for self-reporting.
The Regions considered DEF’s, DEC’s, DECorp’s, and DEP’s compliance history in determining the penalty. The Regions considered that Duke’s relevant compliance history did not serve to aggravate the penalty because although the prior instance involved similar conduct as the instant violation, mitigation activities for the prior instance would not have likely prevented the instant violation from occurring. Additionally, the prior violation involved one instance and was a minimal risk issue filed as a Compliance Exception.
The Regions are awarding some mitigating credit for the level of cooperation demonstrated by Duke. Duke has been highly cooperative with the Regions during the investigation and enforcement processing of the violation by providing the Regions with timely, detailed, and organized evidence in response to Requests for Information. Duke fully cooperated in the Regions’ assessment of the violation and all associated mitigating activities, and openly shared information regarding relevant processes, procedures, internal controls, and organizational oversight of its compliance program. Duke staff and Executives have engaged with the Regions’ staff and Executives throughout the enforcement process by having open and transparent discussions regarding the facts and circumstances of the violation, progress on mitigation, and any potential noncompliance issues identified as a result of the mitigating activities. This is behavior that the Regions highly encourage. Additionally, when the Regions requested field walk-downs to determine the extent of condition, each Duke Company walked-down at least 50 Facilities, well above the minimum required. Further, while only DECorp was required to complete full walk-downs of all of its Facilities based on the initial walk-down findings, DEC, DEF, and DEP voluntarily committed to do the same for their Transmission stations and substation Facilities within the same aggressive timeline used by DECorp. This behavior demonstrates Duke’s level of cooperation with the Regions and overall commitment to reducing risks to the BPS.
Duke agreed to settle the violation resolved by this Agreement, thereby avoiding a hearing on this matter. The Regions applied mitigating credit because it is important to promote prompt resolution of enforcement actions so that Duke’s focus is on mitigation and reducing risks to reliability.
