Methodology to Decrease Packet Loss in IEC 61850 Substation Communication over Wireless 5G Communication

Summary

Wireless networks provide huge potential for power system communications as a flexible and cost-effective alternative. However, based on our previous tests with protection communication have indicated that the IEC 61850 based data packet sizes may not be optimal for wireless communication networks. Processing a vast amount of small data packets might also cause packet loss in the networking devices. To decrease packet loss at the power system communication networks, a methodology of combining the Ethernet packets of multiple Sampled Value messages has been tested in a pilot environment. Our methodology has been inspired by IEC 61869-9, in which there are preferably 2 or 6 Application Service Data Units (ASDUs) per frame. However, our methodology simply aggregates the Ethernet payloads of several packets instead of combining the redundant information on the ASDU level. This aggregation has been executed both vertically by combining consecutive packets and horizontally by combining packets of several merging units. The aggregation has been implemented with Rust low-level network programming language using the pnet library.

The methodology has been validated against wired and wireless reference measurements in our pilot environment. A protection use case is used as a realistic application, in which a virtual fault passage indicator uses directional overcurrent protection to indicate the fault locations. The experimental setup consists of multiple merging units, a smart substation control and protection device operating as an edge device, and a wireless 5G communication network, which is coupled with the HIL simulation at the switches and intelligent network appliances. The packet aggregation is implemented at the intelligent network appliances for both compressing at the sender and decompressing at the receiver. Therefore, the packets received by the substation automation devices are not impacted by the aggregation as it remains visible in the wireless communication network only.

1. Introduction

With the power systems transitioning towards carbon-neutral clean energy systems, the number of connected devices will increase notably. The modern digitalised grids require reliable communication for resilient data exchange. Wireless networks provide huge potential for power system communications as a flexible and cost-effective alternative, especially at remote locations and in retrofit cases. Wireless communication has benefits over wired such as installation without the additional cost of wiring, while also underground maintenance can be avoided. While prior wireless communication, e.g. 2G/3G/4G, can be harnessed for monitoring and control communication, accessing 5G services could have intriguing benefits for power system measurements and protection communication.

In modern power systems, the amount of data transmission is increasing exponentially. While in traditional power systems, there was no need for complex communication infrastructures, these are crucial for modern digitalised power systems and substations. Fast and reliable data sharing enables using a variety of novel power system monitoring, control, and protection applications. The next phase in the modern digitalised substations is coordinated smart protection, which takes into account data streams of several different devices at and around the substation. This in turn will enable redesigning power system monitoring, control and protection architecture, e.g. using edge devices, to be more efficient and closely aligned with communication infrastructure [1].

Traditionally, power systems have relied on wired communication due to their higher reliability over earlier generations of wireless communication. With the introduction of 5G, the reliability of wireless communication is expected to increase enabling it to support the most rigorous power system communication as well. Prior wireless 4G communication has already been used for monitoring and control communication in power systems, but protection communication has such strict requirements that 4G has difficulties meeting them [2]. Developing 5G offers several intriguing features with great potential for power system communications including ultra-reliable low latency communication (URLLC), network slicing, and edge computing. Harnessing these features could enable wireless technologies to be utilised for protection communication and updating power system communication architecture to the 21st century by placing power system edge devices at base stations allowing flexible locating of devices and intelligent aggregated data monitoring.

An overview of substation communication architecture is portrayed in Figure 1a. The measurement transformers and sensors are connected to intelligent electronic devices (IEDs), which transmit data via process bus. IEDs connect to human machine interfaces (HMIs), supervision control and data acquisition (SCADA) and control centre through station bus. At an IEC 61850 based substation, the measurements are transmitted by Sampled Value (SV) communication protocol on layer 2 and similarly the events by Generic Object Oriented Substation Event (GOOSE) communication protocol. SV communication occurs on the process bus and GOOSE on both the process bus and station bus. Our paper is focused on the process bus communication via SV communication protocol.

In modern directional protection communication, merging units are used to supply the measurements to a coordinated IED or edge device compared to IEDs exchanging data bilaterally in a traditional system (see Figure 1bc). Merging units have only the bare minimum functionality, while a majority of the control and protection functionality is located at the coordinated IED. This means that the data output from the merging unit is higher than in the traditional system. For instance, the measurement and status signal data transmitted from the merging unit can be 4 Mb/s as per the requirements in IEC 61850.

According to previous research [3], it has been noted that the data packet sizes based on the IEC 61850 substation automation standard may not be optimal for wireless communication networks. Processing a vast amount of small data packets might cause packet loss in the networking devices. This could be due to several reasons in the communication network. For instance, allocating the data in small packets compared to large increases events requiring processing and thus the probability of packet loss. To combat this issue and decrease the amount of packet loss at the secondary substation communication, a variety of methodologies of combining the Ethernet packets can be used.

For this paper, we have implemented, tested, and verified a methodology for enlarging packet sizes by combining the Ethernet packets either horizontally or vertically. To verify the proposed methodology, we have performed quality-of-service (QoS) measurements in a hybrid experimental setup consisting of hardware-in-the-loop (HIL) simulation, protection devices, and communication networks. This paper is structured by first starting with an introduction, then discussing related work, and our methodology. This will be followed by the description of the experimental setup and a chapter discussing the measurements and validation. The paper will conclude with results and conclusions.

2. Related work

IEC 61850 defines the data model and communication protocols at substations including the SV communication protocol for transferring streams of measurement data in part IEC 61850-9-2. Furthermore, this part, also commonly known as 9-2LE, defines sampling rates and numbers of consecutive measurements per frame. This standard is commonly used on merging units, which sample and digitalise the measurements from the different sensors to be used by the protection, control and monitoring at the digital substations and control rooms.

Methodologies to improve the substation data transmission targeting the 9-2LE especially concerning the inter substation communication have been vastly studied due to the challenging inter substation conditions. Several data compression methodologies have been presented [4-7] especially concerning Phasor Measurement Unit (PMU) waveform transmission. In [8], a compression methodology based on the travelling wave has been suggested to improve the PMU based SV communication at inter substation level. The challenge of high sampling frequency requirements set in IEC 61850 was proposed to be mitigated by modifying the IEC 61850-9-2 SV message format among other solutions to improve the compatibility with travelling wave-based communication [8].

IEC 61850-9-2 has been further advantaged in IEC 61869-9, which defines requirements for digital communication of instrument transformers including the sample rates displayed in Table I. According to IEC 61869-9 recommendations, consecutive sample values can be compressed. This aggregation is done by combining the consecutive measurements, which are called Application Service Data Units (ASDUs), into the same Ethernet frame. With this methodology, the bandwidth can be used more efficiently, but the latency of the measurement sample increases. The latency increase is dependent on the order of the ASDUs as the first ASDU in the Ethernet frame must wait the longest compared to the last ASDU to be added just before the aggregated packet is sent.

The possibilities of IEC 61869-9 have been previously discussed in [10]. It was suggested that the SV packets could be compressed further than is already suggested in IEC 61869-9 by removing the redundant data on the ASDUs to appear only once in the aggregated packet, e.g., “svID”, “smpCnt”, and “confRev” [10]. The compounding impact of this compression methodology is especially visible with the sample rate of 14.4 kHz as in this case 6 ASDUs are being aggregated into the same frame reducing the redundant data to sixth. According to [10], this methodology reduces the size of the redundant information by 60 bytes on each of ASDUs apart from the first ASDU of the frame. However, this methodology requires deep knowledge of IEC 61869-9, which pure ICT engineering personnel of a mobile operator might lack.

Moreover, to compress the SV information and provide it in more efficiency [11, 12] suggest proceeding away from the standardized SV communication protocol and instead using a distributed signal processing unit (DSPU) based approach to combat the required high sampling rates of IEC 61869-9. Essentially, instead of transmitting sampled data via SV protocol, GOOSE communication protocol is used. Typically, GOOSE is implemented to transmit sporadic time-critical data, instead [11] proposes to transmit only periodic data, which reduces the required bandwidth. However, employing periodic GOOSE transmission lacks the possibility for sporadic data transmission of events, which in the case of faults in the system is crucial. Thus, a possibility for alarm and fault signalling must be provided by some other means or the periodic GOOSE communication should be limited to specific devices in the network. Furthermore, GOOSE communication is quite vulnerable to various cyber-attacks, e.g., false data injection attack described in [13] and while IEC 62351-6 has introduced many improvements to authentication and encryption [14], implementing a majority of process bus communication on GOOSE may leave the devices vulnerable.

Our proposed methodology targets mainly secondary substations and other similar units, from which measurement and status data should be collected and transferred to higher levels in the control chain, e.g. control room. Especially in future grids exploiting edge computing and artificial intelligence based monitoring, there is a demand for aggregating data from various points in the grid to coordinated locations such as edge devices. Our methodology offers an option for aggregating the data from different components at the secondary substation, thus only aggregated data optimised for wireless 5G communication would be transmitted in the wireless communication channel decreasing data loss and saving bandwidth.

Compared to the proposed methodologies in the prior research, our methodology does not require changes to the formatting of the SV message packets but rather targets the Ethernet headers of the packets with the data aggregation methodology aiming to improve the transmission of packets in the wireless communication channel by aggregating multiple horizontal or vertical packets under the one Ethernet header. The methodology has been implemented compatible with the standard sampling rates and we have validated the proposed methodology using merging units. Moreover, our methodology intends to remain accessible for ICT engineers with no prior knowledge of 9-2LE and IEC 61869-9. Therefore, the methodology could be widely applied by telecommunication providers without requiring in-depth power system knowledge.

3. Proposed methodology

We propose a data aggregation methodology to improve the performance of the IEC 61850 based communication on a wireless communication channel. This methodology targets the Ethernet headers and proposes to aggregate either horizontal or vertical packets. In the horizontal case, packets from several merging units are aggregated under the same Ethernet header. This would enable packets from multiple merging units to be aggregated and transmitted together to a coordinated collection point via a wireless communication channel. The horizontal case could be directly applied on standard protocols such as 9-2LE or IEC 61869-9 based communication. Correspondingly, in the vertical case consecutive packets are aggregated together from a single merging unit and transmitted together, very much like IEC 61869-9 recommends for 2 or 6 ASDUs in the same frame. The vertical case does not natively support the existing standardised communication but would require increasing the sampling frequency. Our tests have mainly focused on the horizontal case, and we have used the vertical case as a supporting investigation. The aim of applying this methodology is to decrease the amount of packet loss, and especially in horizontal case to decrease the amount of data streams delivered over wireless network, while on the contrary, we hypothesised that the latencies would slightly increase.

In this study, we did the tests with the idea of combining the SV packets in one aggregated packet (see Figure 2). The aggregated packet size, and whether the aggregated packet will include SV packets only from one source (vertical case) or from all sources (horizontal case) can be decided at the time the program is started. The new aggregated packet will be constructed with the TLV (Type, Length, Value) headers, which include the information about the original packets. Type values are as follows:

• type: 1 dst mac address
• type: 2 src mac address
• type: 3 number of packets to be included in this aggregated packet
• type: 4 dst mac address index [0-x]
• type: 5 payload of packet x
• type: 6 src mac address index [0-x]

The aggregated packet includes first TLV value of the original packet count. We have tested several different numbers of original packets in one aggregated packet. Then there is TLVs of destination addresses, first header and then all destination Media Access Control (MAC) addresses in the Value field. Next, there are source MAC addresses of the original packets like in destination addresses following with payloads of the original packets with the TLVs of source MAC index and destination MAC index and the payloads of every packet.

Figure 2 shows how two original SV packets are combined into one aggregated packet in the horizontal case including packets from several sources. In the vertical case, the structure can still be slightly optimised because it includes packets only from one source address. One 3 byte TLV (T=6) can be stripped off. On the other hand, in these specific SV packets, destination MAC addresses are always the same, so also TLV (T=4) might be stripped off if there is a need for more optimisation.

The program was developed using the Rust programming language [15] and using pnet [16] low-level networking library. [17] indicates that Rust language produces maintainable and structured code. The language is also developed with concurrency and parallelism in mind and provides guarantees of isolation and memory safety. [18]

4. Experimental setup description

The experimental setup used in this study fluidly combines testing of communication networks and power systems at the switches and intelligent network appliances, which form common coupling points between the systems. The setup has been described in detail in [3].

The communication network infrastructure of the pilot environment consists of commercial 5G networks, Test Network Finland (5GTNF), emulated wireless connections, and wired connections, which with a versatile QoS measurement system form a platform for testing communication of the power-system data streams under realistic conditions. The test platform enables differentiating the effects between a variety of selected communication technologies and mobile services, routing, configuration, security options, and system processes. Protection and measurement devices are synchronised using IEEE 1588v2 (PTP) time synchronization to two network switches acting as boundary clocks, an atomic clock of the National Metrology Institute of Finland’s (MIKES) serving as a grandmaster. The Quality of Service (QoS) measurements are performed with the passive measurement tool Qosium comprehensively depicted in [2].

The power system of the setup has been implemented as HIL simulation, in which devices under the test are merging units and the smart substation control and protection device as an edge device. A simplistic model of a power system is simulated on a real-time simulator (RTS). The current and voltage measurements from the power system model at the RTS are fed to the merging units via amplifiers. Therefore, the merging units operate as if they would be in a 20 kV grid. Merging units record and digitalise the measurements, which are fed as SV streams to an edge device to operate the protection functions. For the sake of simplicity, we have used a third merging unit to receive the feedback signal on successfully protected faults to the RTS from the edge device. This experimental setup is portrayed in Figure 3a.

In this study, SV messages were transmitted between the merging units and the edge device over an open Virtual Private Network (VPN) connection via a commercial 5G mobile network, Internet, and our test network. The rest of the devices used wired connections. We selected UDP protocol for VPN due to the capability to maintain static data flows, lower jitter, and strict real-time requirements even though the packet loss is typically higher with UDP than with TCP protocol. We built the experimental setup depicted in Figure 3a to confirm that our methodology for packet aggregation can be operated with commercial merging units. However, the used devices only support a 9-2LE compliant sampling rate, thus emulated merging unit SV streams were also implemented on RTS. The modified experimental setup with the emulated merging units is displayed in Figure 3b.

 

5. Measurement and validation

Excluding the measurement system tests and the reference measurements using a fixed RJ45 connection, we performed the measurements of this study in a commercial 5G non-standalone (NSA) network. An industrial outdoor 5G router was placed on the roof of the laboratory building using fibre connections. The distance between the base station and the wireless router was approximately 400 m allowing good signal strength. We performed the test measurements with a common ‘business’ subscription enabling full data transmission capabilities and locking the wireless router to a 3500 MHz 5G cell having an 1800 MHz cell as a 4G anchor.

We validated the data aggregation methodology by measuring the QoS of the SV data streams with different combinations of sampling rates and packet aggregation methodologies described in the upper table (Table II). To compare the results with our methodology, we recorded reference measurements of standard SV data transmission (defined for the 50 Hz system in Table I) as well. Even though some of the sampling rates are not recommended by IEC 61869-9 but retained for backward compatibility with 9-2LE they were still used in the testing to provide more variety to the comparison. We excluded the 96000 frames/s case from the tests due to the high uplink (UL) bandwidth requirement. In addition, we also excluded Case-4b from the tests since our main interest was in the aggregation of small data packets. Data bundling in Case-4b produces data streams of 1619 B packets which would not have been comparable to the other cases due to fragmentation in the network. However, we measured the standard Case-4a to give a wider perspective on the results.

We performed test measurements with both the merging units and SV streams emulated by RTS. A larger set of measurements was conducted with the emulated SV streams as RTS offered more configuration options of different sampling rates, while the commercial merging units were guaranteed proper operation with a sampling rate of 4000 Hz and with 1 ASDU per frame. The test lengths were 2 - 9 h. We performed two to four 2 - 3 h measurements for each test case, and in addition, 9 h measurements for the test cases 1a, 1b, 2a, 2b, 3a and 3b. Depending on the case, each measurement contained 5 - 6 simultaneous QoS measurements. We examined the SV stream 1 in the most detail with three pairs of measurement probes placed along the communication path as depicted in Figure 3. Measurement results for this stream are presented in the following section. SV stream 2 was measured with two probe pairs having consistent results with data stream 1. Since our main interest was in UL communication, we performed additional measurements for SV stream 3 only to find out possible problems in a wireless router and data transmission in the downlink direction.

6. Results

Measurement results for the SV stream 1 are presented in the table below (Table III). The table shows measured packet sizes as well as one-way latency, jitter, and packet loss results produced from the QoS measurement samples that were averaged over 1 s from the packet level measurements. Depending on the test case, each QoS sample contains thousands of individual measurement results, and the total result is tens of millions of individual measurement results. Mean, minimum, and maximum values present averaged values computed from the QoS samples. Peak minimum and maximum values and connection breaks including consecutive packet losses were also recorded.

The measurement results presented in Table III show that compared to the corresponding reference cases the horizontal data aggregation seems to decrease packet loss in cases where packet sizes are small. Unexpectedly, the aggregation does not appear to negatively affect the delays, but on the contrary, the mean and maximum latencies seem to decrease which indicates better performance in the wireless link. Jitter remains low in all cases. In most of the measurements, the horizontal aggregation method also seemed to balance the load at the receiving end. Figure 4 further illustrates the delay and packet loss results in selected measurements. The results indicate that the overall reliability improves if the horizontal packet aggregation method is used in cases where small packets are sent using high data rates.

7. Conclusion

Prior wireless technologies have been unable to reach the strict reliability requirements of protection communication, but 5G services might help in meeting these. However, the packet size originally introduced in IEC 61850 may not be optimal for wireless communication. Therefore, we tested a methodology for packet aggregation, in which individual data packets are aggregated under the same Ethernet header for transmission in the wireless network. We tested the methodology both horizontally and vertically using a HIL setup with the commercial wireless 5G NSA network. The initial test results indicate that aggregating small packets could decrease the packet loss in the wireless communication channel, but further statistical measurements would be required to confirm the results. Packet aggregation addressing the Ethernet headers can be implemented by communications engineers without extensive knowledge on IEC 61850. Therefore, this methodology could be also used on other verticals and is not limited to power systems. In future research, we intend to test the packet aggregation with 5G services, e.g. network slicing, and communication over 5G SA networks.

References

1. S. Borenius, M. Lehtonen, and J. Costa Requena "Software Defined Grid" (Grid of the Future Symposium, November 2019)
2. P. Hovila et al. "5G networks enabling new smart grid protection solutions" (25th International Conference on Electricity Distribution, June 2019)
3. P. Hovila, H. Kokkoniemi-Tarkkanen, S. Horsmanheimo, P. Raussi, S. Borenius, and K. Ahola "Cellular Networks providing Distribution Grid Communications Platform" (26th International Conference on Electricity Distribution, September 2021)
4. B. Yunus and H. Li "Analysis of PQ waveform for optimum data transmission over IEC 61850 communication standards" (2008 IEEE 2nd International Power and Energy Conference, December 2008, pages 662-665)
5. F. Zhang, L. Cheng, X. Li, Y. Sun, W. Gao, and W. Zhao "Application of a Real-Time Data Compression and Adapted Protocol Technique for WAMS" (IEEE Transactions on Power Systems, 2015, pages 653-662)
6. B. A. Bhuiyan, M. W. Absar, and A. Roy "Performance comparison of various wavelets in compression of PMU generated data in smart grid" (2017 3rd International Conference on Electrical Information and Communication Technology (EICT), December 2017, pages 1-6)
7. S. Acharya and C. L. DeMarco "Low-loss image-based compression for synchrophasor measurements" (IET Generation, Transmission & Distribution, 2020, pages 4571-4579)
8. N. H. Ali, B. M. Ali, M. L. Othman, and K. M. Abdel-Latif "Performance of communication networks for Integrity protection systems based on travelling wave with IEC 61850" (International Journal of Electrical Power & Energy Systems, 2018, pages 664-675)
9. IEC 61869-9 Instrument transformers (Internaltional Electrotechnical Commission, 2016)
10. S. M. Blair, A. J. Roscoe, and J. Irvine "Real-time compression of IEC 61869-9 sampled value data" (2016 IEEE International Workshop on Applied Measurements for Power Systems (AMPS), September 2016, pages 1-6)
11. F. Hohn and L. Nordstrom "Data Models and Protocol Mapping for Reduced Communication Load in Substation Automation with High Sampling Rate Protection Applications" (2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), October 2018, pages 1-6)
12. F. Hohn, T. Rabuzin, J. Wang, and L. Nordström "Distributed signal processing units for centralised substation protection and control" (The Journal of Engineering, 2018, pages 1223-1228)
13. V. S. Rajkumar, M. Tealane, A. Ştefanov, and P. Palensky "Cyber Attacks on Protective Relays in Digital Substations and Impact Analysis" (2020 8th Workshop on Modeling and Simulation of Cyber-Physical Energy Systems, April 2020, pages 1-6)
14. S. M. S. Hussain, T. S. Ustun, and A. Kalam "A Review of IEC 62351 Security Mechanisms for IEC 61850 Message Exchanges" (IEEE Transactions on Industrial Informatics, 2020, pages 5643-5654)
15. "Rust" (https://www.rust-lang.org/ (accessed 8.12.2021))
16. "Crate pnet" (https://docs.rs/pnet/latest/pnet/ (accessed 8.12.2021))
17. L. Ardito, L. Barbato, R. Coppola, M. Valsesia "Evaluation of Rust code verbosity, understandability and complexity" (PeerJ Computer Science, 2021)
18. N. D. Matsakis and F. S. Klock "The rust language" (2014 ACM SIGAda annual conference on High integrity language technology, October 2014, pages 103-104)

---

• Contact Author: P. RAUSSI