Enhancing existing line differential relay operation reliability with hitless IP/MPLS multipath redundancy

Summary

Utilities have long relied on line differential protection systems (87L) to ensure grid safety and integrity. These systems impose stringent communication requirements, including low latency, symmetric delay, jitter mitigation and rapid failure recovery. The longevity of these protection systems means that multiple generations of communication interfaces (e.g., E&M, RS-232, G.702 and IEEE C37.94) often coexist within the same grid infrastructure. Since the life cycle of protection systems typically outlasts that of telecommunication equipment, this creates additional operational challenges. As utilities transition from obsolete SDH/SONET networks to modern packet networks, it can be difficult to maintain the same network service level for relay communications. This transition also presents an opportunity for utilities to add novel network capabilities to enhance the robustness of their existing protection systems.

This paper explores how IP/MPLS networks can bring highly reliable communications to these critical systems through advanced capabilities such as deterministic quality of service (QoS), flexible playout buffers, asymmetrical delay control and redundancy protection. It also describes how IP/MPLS networks can provide full interoperability with multiple generations of relay interfaces, enabling power utilities to extend the lifecycle of their protection systems.

The paper examines key challenges and describes tailored IP/MPLS capabilities for addressing latency, jitter, delay symmetry and hitless recovery needs. It introduces an active multipath mechanism scheme called active multipath pseudowire (AMP), which extends pseudowire redundancy by replicating traffic over multiple active paths to enable zero fault recovery time with multi-fault resilience. The paper also includes the results of an industry validation effort that demonstrates how modern IP/MPLS networks can deliver the utmost reliability and availability required for line differential protection, ensuring grid resiliency and availability

1. Introduction

Power services are foundational to societies in the era of electrification. Line differential protection systems play a critical role in ensuring that the grid can deliver power services safely and reliably. These systems depend on reliable communications between differential relays, which exchange current measurements at the different terminals within the protected zone. Over time, relay interfaces have progressed from analog E&M signals to digital interfaces such as RS-232, G.703, and more recently to the C37.94 optical interface. Despite this evolution, the core functions of the relays remain the same: calculating the differential current, detecting faults and issuing trip commands to isolate them.

2. Deployment life cycle challenges

As line differential protection systems progress through their life cycle, utilities encounter three key challenges:

1. Communication network technology evolution: The transition from time-division multiplexing (TDM) to packet-based networks has rendered SDH/SONET network equipment and support obsolete. Legacy TDM chipsets are no longer produced as semiconductor foundries shift their focus to making advanced chipsets for more lucrative markets such as artificial intelligence (AI) and automobiles. This creates an urgency for utilities to modernize their existing TDM networks.
2. Cost constraints: Replacing different generations of legacy protection systems with new generations of differential relays that support IEC 61850 Generic Object Oriented Substation Event (GOOSE) and Sampled Values (SV) protocols is very often labor intensive, and requires meticulous planning. It often also requires utilities to install merging units and optical fiber in the switchyard, which further complicates projects.
3. Operational continuity: It is imperative that operational continuity be maintained for maximal operational efficiency.

Since migrating from SDH/SONET networks to packet networks is inevitable, the packet network must be able to continue supporting line differential relay communications with the same service characteristics. This paper discusses the stringent network requirements for relay communications and explores how IP/MPLS capabilities can address these challenges. It also introduces a novel IP/MPLS network service capability that surpasses the recovery performance of legacy SDH/SONET networks by providing 0 millisecond (ms) recovery speeds.

3. Network challenges

Of all critical grid applications, line differential protection has the most stringent network requirements.

3.1. Connecting legacy communication interfaces

Line differential relays employ a diverse range of of legacy interfaces, requiring the use of network node with comprehensive interfaces support to ensure full communication interoperability.

3.2. Latency

Line differential protection systems must clear a given fault within a predetermined critical clearing time. High network latency can delay relay operations and increase fault clearing time beyond this critical threshold. This can compromise public safety, cause excessive asset wear and degrade power quality. The IP/MPLS network must ensure low latency at all times and in all conditions.

3.3. Jitter

Most relay communications are sent in TDM formats such as IEEE C37.94. These TDM traffic streams are transported in MPLS packets using TDM pseudowires [1] that are sensitive to jitter, also known as delay variation. Jitter can cause TDM pseudowires to enter buffer overrun or underrun states.

3.4. Delay asymmetry

Delay asymmetry refers to the difference in latency between the forward and reverse paths. While delay asymmetry does not impair TDM pseudowires, line differential relays are highly sensitive to it. This is because they depend on the “ping-pong” protocol to measure the total round-trip delay during system startup and divide the result by two to estimate the one-way propagation delay. Precise estimation of this one-way delay is crucial for time-aligning current measurements from different terminals. Delay asymmetry that occurs after the ping-pong process has ended would result in inaccurate delay estimation, causing the relays to falsely issue trip commands.

3.5 Network failure recovery

It is critical that relay communications are restored in the shortest possible time when a network failure occurs. If the network breaks and relay communication stops, relay operations cease, leaving the grid vulnerable to line faults. Recognizing the importance of uninterrupted relay operations, IEC 61850-90-12:2020 [2] classifies line differential protection in the most stringent 0 ms recovery delay class. This is an extremely challenging requirement that even SONET/SDH technologies, often considered the benchmark for network reliability, fail to meet. ANSI T1.105.01-2000 specifies a switchover time within 50 ms. [3]

3.6. Multi-fault resiliency

As atmospheric events become more frequent and intense, multi-fault failure scenarios are an increasingly common occurrence. As a result, multi-fault resiliency has become a fundamental requirement for grid communications in general and differential relay communications in particular.

4. Tackling relay communications challenges with IP/MPLS

This section explains how IP/MPLS capabilities can meet the stringent requirements for relay communications, with a focus on redundancy protection.

4.1. Full interoperability with legacy communication interfaces

Line Differential Protection relays deployed today most commonly use ITU-T G.703 [4], IEEE C37.94 [5], ITU-T X.21 [6] or E&M [7] communication interfaces. IP/MPLS routers need to support this wide range of interfaces. They packetize traffic coming from the interfaces onto MPLS frames and transport it in TDM pseudowires.

4.2. Deterministic quality of service

With robust traffic management mechanisms such as classification, queuing and scheduling, IP/MPLS supports deterministic QoS to ensure that relay communications are always serviced with the highest priority and lowest latency (Figure 1).

4.3. Flexible playout buffer

Jitter, or delay variation, is unavoidable even in well-engineered networks that have stringent QoS because of the statistical multiplexing nature of packet switching. A low-priority long packet (1,500 bytes or longer) in transmission can temporarily block relay communications in the highest priority queue, a phenomenon called head-of-line (HOL) blocking. This extra delay can cause the receiving router to be starved of TDM traffic to send to the relay, which causes an alarm at the TDM interface.

To mitigate the impact of jitter, a playout buffer in the IP/MPLS edge router that terminates the pseudowire holds incoming packets for a certain period of time before forwarding them to the relay. This smooths out variable network delays. Configuring the buffer size according to jitter in the network allows the router to neutralize the jitter while optimizing the delay. A buffer size of n ms means the TDM traffic will play out when n/2 ms of buffer is filled, meaning the buffer would incur a delay of n/2 ms. Deployment experience shows that a buffer size ranging from 6 to 10 ms would suffice when an appropriate QoS policy is applied in the network. The buffer configuration typically contributes 3 to 5 ms to the overall communication delay budget, which is generally within the range of 5 – 15 ms. It should be noted that communication delay is just one component of the overall fault clearing time budget, which also includes fault detection time (typically 20 – 50 ms) and breaker opening time (typically 30 – 80 ms).

4.4. Asymmetrical delay control

Accurate estimation of one-way propagation delay in the network is pivotal for ensuring reliable line differential protection. Although the forward and reverse relay communication paths follow the same route, delay asymmetry can still occur during the buffer “priming” stage, when the TDM data stream is initiated [8]. At this stage, as the initial packetized TDM stream traverses the network in both directions, jitter in the network can add unequal delays in each direction. Having delay asymmetry early on results in the buffers at both ends “playing out” TDM streams with inconsistent buffer residence times (i.e., the duration between playout time and arrival time) for the forward and reverse paths. This is the cause of asymmetrical delay.

A smart playout buffer capability called asymmetrical delay control (ADC) runs on both ends of the TDM pseudowire to remedy this asymmetrical delay. During pseudowire startup, an ADC analysis and adjustment stage occurs. It determines the buffer residence time for a large number of packets (e.g., 4,000 packets). At the end of the measuring period, ADC calculates an average residence time. It then adjusts the buffer by adding or removing bytes to match the “engineered” delay, restoring delay symmetry (Figure 2).

4.5. Active multipath pseudowire

TDM pseudowires support redundancy as a standard capability [9]. Pseudowire redundancy provides protection against network failures by establishing redundant active and backup paths between the IP/MPLS routers at both ends of the pseudowire. When an active path failure is detected, traffic is switched to the backup path. This minimizes communication disruption. Crucially, by harnessing traffic engineering with explicit paths, the backup path is provisioned over a diverse route to eliminate shared risk link groups (SRLGs) with the primary path.

Pseudowire redundancy models a SONET/SDH APS 1+1 scheme with a target recovery time of 60 ms. While this provides sufficient communication protection for many critical applications, including SCADA and synchrophasors, a 60 ms breakdown in relay communications will expose the grid to line faults, particularly if a weather event simultaneously impacts the transmission line and the communication link.

Active multipath pseudowire, or AMP, is an innovative extension of pseudowire redundancy. This approach is modeled on the Parallel Redundancy Protocol (PRP) used in the IEC 61850 process bus [9], where an intelligent electronic device (IED) replicates traffic over two parallel local area networks (LANs). In this case, the IP/MPLS router packetizes data received from the relay and replicates and transmits the MPLS packets over two active pseudowires in one IP/MPLS network. The router on the other end receives both copies from these two pseudowires, selects the first received copy and forwards the data contained in the packet to the relay. It should be noted that these two pseudowires ride over strict label switched paths (LSPs) that follows exact, hop-by-hop routes, and are not subject to fast re-route (FRR) mechanism.

Below is a step-by-step walk-through of the protection mechanism for the forward path (Figure 3). The reverse path steps are identical.

1. The relay sends data to the connected router over a TDM interface such as IEEE C37.94.
2. The router packetizes the TDM data stream into MPLS packets, each of which contains sequence numbers.
3. The router replicates the packets onto two active paths (green and blue).
4. The replicated traffic traverses the network.
5. The traffic over the green path arrives earlier than the blue path because there are fewer hops.
6. A smart combiner uses decision logic to analyze the sequence numbers embedded in the packets and select the early-arriving copy from the green path for buffer playout while ensuring proper traffic ordering.
7. The playout buffer sends data to the other relay over a TDM interface.

With this protection mechanism, all paths are active at all times, in contrast to standard redundancy protection where only one path is active. Furthermore, unlike SONET/SDH APS 1+1 protection where the receiving node only listens to the active link, the receiving IP/MPLS routers listen to data streams from all active paths concurrently. When there is a failure (node or link) along the green path, network recovery is hitless. The receiving router continues receiving the replicated packets from the blue path and forwards the TDM data to the relay.

Even though the blue path has a higher latency than the green path, the end-to-end delay symmetry is still maintained because the difference is compensated by a lower delay associated with jitter buffer fill. As the playout buffer waits for the packet with the next expected sequence number to arrive from the slower blue path, the jitter buffer fill is reduced as packets are removed to be played out back to TDM based on the network clock. As a result, the total of path delay and jitter buffer delay remains the same for the usage of either path. Relay operations are uninterrupted and the relays continue to communicate with each other, with no awareness of a network failure.

4.6. Multi-fault resiliency

Multi-fault resilience is pivotal for attaining the highest level of availability for relay communications. To address the need for higher resilience, this mechanism is further extended to support more than two active paths. The transmitting router replicates traffic onto all the paths. The receiving router listens to all the streams and selects the earliest-arriving copy.

A network with rich and diverse physical routes can benefit from this active multipath pseudowire capability. As in the case of two active paths, by harnessing IP/MPLS engineering and explicit path capabilities, each active path can be provisioned on diverse routes to eliminate SRLGs and attain multi-fault resilience.

In the example deployment shown in Figure 4, four active paths follow diverse routes. Network failures are affecting active paths #1, #2 and #3, but active path #4 is still up and running. The receiving IP/MPLS router continues to receive packets from active path #4 without any impairments and forwards the data in the packets to the relay. Delay asymmetry is again maintained throughout if any path fails. When failed paths recover, the packets they carry may be immediately selected by the receiving routers without any impact on delay symmetry.

4.7. Interaction with ADC

In this active multipath redundancy protection scheme, packets arriving at the playout buffer can come from any active path and each path can have a different delay characteristic. As a result, special attention needs to be paid to the required playout buffer size.

If there are multiple paths, additional care must be taken during the ADC analysis and adjustment stage at TDM pseudowire startup. Only paths with both traffic directions up and running are considered available and to be used for analysis.

In scenarios where no protection scheme is deployed, the buffer size is configured to absorb the maximum jitter the MPLS packets can experience in the network. With active multipath redundancy, the playout buffer size also needs to accommodate the difference between the shortest delay and the longest delay. The optimal engineering rule of thumb is to set it to twice the sum of the maximum network jitter and delay difference between the fastest and slowest paths.

5. Validation

Extensive research and practical analyses have validated the use of IP/MPLS for transporting line differential traffic over the years [11][12][13][14][15]. This paper builds upon this foundation by validating a novel redundancy approach. Nokia has tested this redundancy protection mechanism in collaboration with power utilities and industry partners [16][17]. The test setup included differential relays single-connected to an IP/MPLS network with C37.94 interfaces. The network provided two active, diverse paths that connected a pair of relays.

One key focus area of the tests was to investigate the impact of network link and node failures on relays. The tests demonstrated that with active multipath redundancy protection configured, failures induced to impair one path produced no errors recorded by the relay and had no impact on the line differential protection system’s performance. Together with other test cases, the validation efforts successfully demonstrated that IP/MPLS networks can meet the stringent network requirements of differential relay communications, including 0 ms recovery time.

6. Conclusion

Our study introduces a novel IP/MPLS network redundancy protection mechanism that achieves the ambitious goal of 0 ms network recovery for line differential protection. This meets the stringent requirements in IEC TR 61850-90-12:2020.
Our mechanism comprises two critical aspects:

1. Hitless protection: By using an active multipath approach, our mechanism ensures uninterrupted relay communication even during multi-fault scenarios within the network.
2. Persistent asymmetrical delay control: Through seamless integration with ADC, delay symmetry is maintained during protection recovery.

This protection mechanism can provide utilities with much greater confidence in migrating line differential relay communications to IP/MPLS networks, enabling them to extend the lifespan of existing legacy line differential protection systems.

References

1. Vainshtein, A. and Y. Galtzur, "Layer Two Tunneling Protocol version 3 - Setup of Time-Division Multiplexing (TDM) Pseudowires", IETF RFC 5611, August 2009.
2. IEC TR 61850-90-12:2020, Communication networks and systems for power utility automation - Part 90-12: Wide area network engineering guidelines, Edition 2.0, International Electrotechnical Commission, Geneva, Switzerland, 2020.
3. ANSI T1.105.01-2000, Synchronous Optical Network (SONET) – Automatic Protection, 2000.
4. ITU-T G.703-2001, Physical/electrical characteristics of hierarchical digital interfaces, 2001.
5. IEEE C37.94-2017, IEEE Standard for N times 64 kbps Optical Fiber Interfaces between Teleprotection and Multiplexer Equipment, 2017.
6. ITU-T X.21, Interface between Data Terminal Equipment and Data Circuit-Terminating Equipment for synchronous operation on public data networks, 1993.
7. ETSI, Access, Terminals, Transmission and Multiplexing (ATTM); Integrated Broadband Cable and Television Networks; IPCablecom 1.5; Part 15 Analog Trunking for PBX Specification (ETSI TS 103 161-15).
8. S. M. Blair et al., "Validating secure and reliable IP/MPLS communications for current differential protection," 13th International Conference on Development in Power System Protection 2016 (DPSP), Edinburgh, UK, 2016, pp. 1-6.
9. Muley, P., Aissaoui, M., and M. Bocci, "Pseudowire Redundancy", IETF RFC 6718, August 2012.
10. IEC 62439-3:2021, Industrial communication networks - High availability automation networks - Part 3: Parallel Redundancy Protocol (PRP) and High-availability Seamless Redundancy (HSR), 4th Edition, International Electrotechnical Commission, Geneva, Switzerland, 2021.
11. Blair, S.M., Guo, H., Booth, N., Nawacki, N., Hite, L., Verhulst, D., “Comprehensive Validation of Packet-based Communications for Future Energy Systems”, CIGRE Session 2020, Ref D2-304_2020.
12. Jahr, A., Gorbing, A., Schneider, S., “Line differential protection communication via MPLS-based inter substation communication networks”, CIGRE, 2019.
13. Shahi, M., Tuzon, P., Sinclair, M., Gharti Chhetri, G., “Teleprotection signal testing over IP/MPLS network”, CIGRE Session 2022, Ref D2-10589_2022
14. Tan, V., Cole, J., “Teleprotection over Multiprotocol Label Switching (MPLS): Experiences from an Australian Electric Power Utility”, CIGRE Session 2018, Ref D2-305_2018
15. Blair, S.M., Booth, C.D., de Valck, B., Verhulst, D., “Demonstration and analysis of IP/MPLS communications for delivering power system protection solutions using IEEE 37.94, IEC 61860 Sampled Values, and IEC 6150 GOOSE protocols”, CIGRE Session 2014, Ref B5-111_2014 
16. Louh, A., Goerbing, A., Verhulst, D., Loef, R., Royle, K., Stachel, P., Blumschein, J., "Implementing IP/MPLS network-based synchronization for line differential protection and control", CIGRE Science & Engineering – No31, December 2023.
17. Subieta, M., Chan, H., Wright, R., Santin, E., “Failover Redundancy Mechanism Ensuring Hitless Communications for Line Differential Protection”, CIGRE 2024 Grid of the Future Colloquium.

---

• Contact Author: D. VERHULST