During the July 2024 incident, a 230 kV transmission line fault led to customer-initiated simultaneous loss of approximately 1500 MW of voltage-sensitive load that was not anticipated by the BES operators. The electric grid has not historically experienced simultaneous load losses of this magnitude in response to a fault on the system, NERC notes. The grid has historically been planned to deal with large generation losses but not for such significant simultaneous load losses.
Simultaneous large load losses have two effects on the electric system, NERC says. First, frequency rises on the system as a result of the imbalance between load and generation. Second, voltage rises rapidly because less power is flowing through the system. In the July 2024 incident, the frequency did not rise to a level high enough to cause concern. The voltage also did not rise to levels that posed a reliability risk, but operators did have to take action to reduce the voltage to within normal operating levels. However, as the potential for this type of load loss increases, the risk of frequency and voltage issues also increases, NERC observes, and operators and planners should be aware of this reliability risk and ensure that these load losses do not reach intolerable levels.
The incident in more detail
At 7 pm Eastern on 10 July 2024, a lightning arrestor failed on a 230 kV transmission line in the Eastern Interconnection, resulting in a lasting fault that eventually “locked out” the transmission line. The auto-reclosing control on the transmission line was configured for three auto-reclose attempts staggered at each end of the line. This configuration resulted in six successive system faults in an 82-second period. The protection system detected these faults and cleared them properly. The shortest fault duration was the initial fault at 42 milliseconds, and the longest fault duration was 66 milliseconds. The voltage magnitudes during the fault ranged from 0.25 to 0.40 per unit in the load-loss.
Coincident with this six-fault disturbance, the same local area saw an approximate 1500 MW of load reduction. See Figure 1. None of this load was disconnected from the system by utility equipment; rather, the load was disconnected on the customer side by customer protection and controls. It was determined that the 1500 MW of load reduction was exclusively data centre type load. The area where the disturbance occurred has a high concentration of data centre loads.
Frequency and voltage rose due to the load loss. See Figures 2 and 3. Frequency rose to a high of 60.047 Hz and settled back to 60.0 Hz in about 4 minutes. At the highest level, voltage rose to 1.07 per unit. Operators removed shunt capacitor banks in the local area to return voltages to normal operating values.
Causes of load reductions
NERC had discussions with data centre owners/operators to understand the specific cause of their load reductions. NERC determined that the data centres transferred their loads to their backup power systems in response to the disturbance. Data centre loads are sensitive to voltage disturbances. The data centre protections and controls are designed to avoid equipment outages in the event of voltage disturbances.
In addition to the computer equipment at these facilities, cooling equipment is also critical to the operation of the data centre and sensitive to voltage disturbances. To ride through voltage disturbances on the electric grid, data centres employ uninterruptible power supply (UPS) systems that will instantaneously take over providing power to the data centre equipment when a grid disturbance occurs.
The differing types and designs of these UPS systems cause differences in the characteristics of the data centre responses to a voltage disturbance.
A centralised design typically uses UPS systems at the load centre level that are in the range of 2–5 MW. The UPS uses power electronics to switch the load to a battery bank connected to the UPS. These battery banks are not designed to supply the load for long periods but rather to power the load for the relatively short duration of a disturbance, or — in the case of a complete electric grid outage — long enough to start a backup generator that will then provide power to the UPS.
A decentralised UPS design uses many smaller UPS systems at the rack level. These rack-mounted UPS systems are typically in the range of 3–4 kW.
Decentralised UPS systems operate in a similar way to the centralised UPS, just on a smaller scale.
Another type of UPS is a dynamic/diesel rotary uninterruptible power supply (DRUPS). See Figure 4. These systems use a flywheel to provide uninterruptible power and a clutch system to quickly start and connect a diesel engine upon encountering a disturbance in the electric grid.
Figure 4. Rolls-Royce mtu DRUPS (Kinetic PowerPack) (source: Rolls-Royce mtu)
The load characteristics of these types of UPS systems differ in response to a transient disturbance on the electric grid.
For the static centralised and decentralised UPS systems that utilise batteries, the load will be taken over by the battery when a transient voltage disturbance occurs. Since it is a transient disturbance, such as a temporary fault on the electric grid, the grid voltage will typically return to normal in milliseconds.
Once the grid voltage returns to normal, the load will then be transferred back to the grid.
Upon detecting a transient voltage disturbance, a DRUPS system will immediately transfer the load to the flywheel/ac generator and start the engine, which will act as the prime mover for the generator before the flywheel exhausts its kinetic energy.
This system will not quickly transfer the load back to the grid after the transient disturbance has cleared and the grid voltage returns to normal. Typically, transferring the load back to the grid from the DRUPS system must be done manually.
Summarising, the typical static UPS system load characteristic, as seen by the grid, is a short-duration loss of load that returns quickly after the transient disturbance clears. In contrast, the typical DRUPS system load characteristic, as seen by the grid, is a loss of load that does not return quickly after the transient disturbance clears.
NERC’s discussions with the data centre owners/operators also identified another protection/control scheme that impacts the response of data centre load to voltage disturbances on the grid. This scheme detects and counts voltage disturbances on the grid. If a certain number of voltage disturbances are seen within a given time, the data centre will transfer its load to the backup system, and it will remain there until it is manually reconnected to the grid. The typical number of voltage disturbances that trigger this scheme is three, and a typical time is one minute. As such, three voltage disturbances within one minute will result in data centres using this protection/control scheme, transferring their load off the grid and staying off until they manually transfer back. This scheme can be deployed with both centralised and decentralised UPS designs.
While the three load characteristics described above were predominant in the July 2024 event, many variations in load characteristics exist. These characteristics are determined by the numerous vendor-supplied controls within a data centre, including vendor-specific UPS controls. Additionally, controls under the purview of data centre owner/operators, such as responding to a particular number of disturbances within a certain period, also determine the characteristic response to system disturbances.
In the case of the July 2024 incident, most of the sustained load reduction occurred simultaneously with the third voltage depression, which coincided with the third automatic reclosing attempt. At that time, approximately 1260 MW of load dropped off the electric grid and did not return for hours. Most of the load loss in this event can be attributed to the interaction between the automatic reclosing sequence on the faulted transmission line and a data centre protection/control scheme that counts the number of voltage disturbances within a specified period.
While the July 2024 incident did not present any significant issues in terms of the reconnection of the loads, the potential exists for issues in future incidents if the load is not reconnected in a controlled manner.
Significant amounts of load being reconnected to the system presents challenges to balancing authorities and transmission system operators. Ramp rates for load connection are just as critical to system operations as generation ramping, says NERC.
Understanding load
The July 2024 incident has also highlighted potential reliability risks to the Bulk Electric System with respect to the voltage ride-through characteristics of large data centre loads. Similar incidents have occurred in other US Interconnections with cryptocurrency mining loads as well as oil/gas loads. While these loads are different from the data centre loads of the July 2024 incident, they present the same challenges to operators and planners of the Bulk Electric System. Understanding the changing dynamic nature of load is critical to its future operation, concludes NERC.
