B5 - Testing approach for RTE’s R#SPACE Protection Automation and Control System

Summary

In the past two decades, the policy of RTE has been to purchase "turn-key" integrated Protection Automation and Control Systems (PACS) from several vendors, based on a set of specifications. In these systems, the vendors were responsible for the associated integration tests. RTE has now launched a project to develop the R#SPACE system, aiming at an industrial rollout of fully digital, IEC 61850 based multi-vendor PACS.
During the design and definition phase of the project, the choice has been made to purchase certain components of the R#SPACE PACS and to develop internally others. A dedicated acquisition strategy for every purchased component has been implemented. This makes R#SPACE a multi-vendor and multi-component system by construction.
Consequently, RTE is in charge of the design of the system, which is based on an interoperability framework, including the configuration, data models and cybersecurity. These items are critical and constitute the core of R#SPACE systems. For this reason, they are directly developped by RTE and not outsourced. This also means that a single actor needs to be mandated to perform the overall integration testing of all components. For the first set of R#SPACE substations, RTE has taken this role of system integrator. In addition, RTE is in charge of developing the tests of the functions of the components developed internally and of the partial integration tests between components. The qualification tests of externally developed components are the responsibility of the manufacturers, with an acceptance test realized by RTE. The FAT is performed by a specific contractor (called “Systemier”) based on test plans and test systems provided by RTE.
This paper gives an overview of the different tests and the associated approaches and tools implemented for R#SPACE. This includes:

• Conformance tests of components developed by RTE
• Conformance tests of components developed by contractors
• Interoperability “pre-integration” tests involving a limited number of components
• End-to-end integration tests covering the complete PACS
• Factory Acceptance Tests (FAT)
• Installation tests, Site Acceptance Tests (SAT) and commissioning tests
• Tests performed on the PACS after commissioning

Each of these items has specific needs for test tools, configuration files (for different functional variants), test scenarios and test scripts. Each of these items has also its specific development and testing schedule. The overall project level approach to coordinate these aspects is described in this paper. This includes the development of an Integration Test Platform, the coordination of the different test tools, the management of constraints related to the configuration of the components in the different testing phases, the tests for functions hosted in a virtualized infrastructure, and the management of the planning and of the scope of the pre-integration and integration testing programs.
Different development schedules cause certain components or their updates to be available at different times in the overall project schedule. The testing management must therefore maintain a global overview of the tested and qualified interoperability between the different hardware and firmware versions of components. These aspects are described at the end of the article, together with other testing issues encountered during the project development.

1. Description of R#SPACE PACS

The R#SPACE system can be divided in several blocks, as shown in Figure 1:

• The substation level functions are implemented in a virtualized infrastructure providing redundancy and hosting the virtual machines which run the substation level applications. These applications include the HMI for operating and maintaining the system, the substation level automations, the gateways, and the cybersecurity functions. The virtualized infrastructure also hosts administration functionalities (IED Configuration Tools (ICT) for instance).
• The components hosting bay and process level functions consist of process interface IEDs: SCU (Switchgear Control Unit) and SAMU (Stand Alone Merging Unit), some of which are redundant; and fully digital IED BCU (Bay Control Unit) hosting bay level automations and protection functions. The bay level may also contain IED hosting specific protection functions which are not fully compliant with the R#SPACE interoperability framework.
• The Local Area Network (LAN) connects the IEDs and the servers hosting the virtualized functions with redundancy (PRP) and provides the time synchronization architecture.

The design is based on an interoperability framework, which includes the IEC 61850 data model [1], in addition to specifications regarding configuration capabilities, communication features or cybersecurity requirements. Some of the devices are purchased, and others, such as substation level automatons, are developed internally.

2. Testing process

2.1. Global overview

The R#SPACE PACS is a multi-supplier system which combines components developed externally with components developed by RTE. Testing activities are performed both by suppliers and by RTE. A specific team of the project (technical lot 9 in Figure 1) is dedicated to coordinating all test activities and qualifying the complete system. Each stage of testing is associated to a set of expected tests, but also to specific stakeholders, tools, and testing environments. It is essential to get an overall vision of testing, and to ensure the completeness of the system tests.

To set up the above organization, several test environments are required. To facilitate the exchange and sharing of expertise, most of these environments are grouped together in a test platform implemented in an RTE site located near Lyon, called Transfo. A few platforms associated to specifics skills or tools are installed in supplier’s facilities.

See a larger image

Although these environments are managed independently, a local network allows to connect the different environments, if required for specific tests. The aim of the different platforms with dedicated test environments to each component and stage of the test process is to identify problems in advance and to reduce risk of late detection of issues for the final system qualification stage.

The following sections describe the tools and methods used in these specific test environments

2.2. Conformance tests in development stage

In development stages, conformance tests of the devices are performed by suppliers (externally, or internally for the IEDs developed by RTE) for each component of the system:

• The Communication network (environment 1a and 1b in Figure 3) is based on a PRP architecture, with separate operational and administration networks. The corresponding conformance tests are related to the validation of the switches, and the time synchronization (Precision Time Protocol PTP).
• The IT virtualization infrastructure (environment 1Va and 1Vb in Figure 3) is tested to verify the virtual network, its extension between hypervisors and its connection to the physical LAN; the IT and cybersecurity services hosted on virtual machines for some operation use cases; and the redundancy mechanisms of the virtualization infrastructure.
• Bay level IEDs are mainly purchased by RTE on the basis of specific requirements. As each supplier is responsible for the compliance of their products with the function and system requirements, RTE made the choice to carry out these tests on the supplier premises, based on a mutually agreed test protocol (Environment 23a in Figure 3). In addition, specific functions, such as distance protection for instance, are also tested by RTE experts on RTE platforms (Environment 23b in Figure 3).
• Substation level automations are developed by RTE and implemented in virtual IED, using an automation development platform (a Soft-PLC platform) that includes an IEC 61850 communication stack. Conformance tests focus on IEC 61850 common components (control mechanism, quality management for instance) and automation algorithms specific to each virtual IED. As the IEC 61850 stack has a conformance certification with IEC 61850 standard, it is not specifically tested by RTE.
• SCADA HMI, developed by RTE on the basis of an off-the-shelf SCADA supervisor, provides the interface of the PACS for operation and maintenance. It also plays an important role in advanced diagnostics and cyber-security. On one hand, the HMI is tested to ensure that the configured SCADA conforms to the imported SCD file, and that the thousands of IEC 61850 data are correctly managed. On the other hand, the HMI can also be actively used to support other R#SPACE tests.
• RTE control centers use IEC 60870-5-104. The Telecontrol Gateway (GW) is developed by a supplier to interface the R#SPACE site with remote control centers using this protocol. Conformance tests are performed step by step using an IED tool simulator to generate IEC 61850 data flows, and a SCADA center emulator to simulate the remote center.

One of the constraints encountered during conformance tests is related to the dependency between components of the system that can have different development schedules. For instance, the availability of the R#SPACE IEDs that comply with all the communication and synchronisation requirements is a condition for the validation of the LAN and the clock server. Also, IT infrastructure conformance tests require the availability of a representative configuration of virtualization servers and the availability of the wide area network (WAN) access for communications with datacenter applications.

Thus, it quickly became necessary to purchase or develop simulators to emulate a real R#SPACE system environment: network traffic simulator (GOOSE, SV, IP traffic), IEDs communication emulator reflecting the real data model of the device, or IEC 60970-5-104 simulator for the gateway.

As it was sometimes required to adapt the product to the project needs or to an evolving technical perimeter, continuous unitary and interoperability tests are essential to the success of the project. Some of the unitary tests are executed using scripting languages with a Read-eval-print loop (REPL) interpreter (like python or lisp) on the top of open-source IEC 61850 stack. This in-house simulation toolkit enables RTE to produce scripted and manual test cases taking into account the context, and to be more efficient during the conformance testing stage.

It was also necessary to provide SCD files for the unitary qualification of components. This concerns both the qualification of the corresponding ICT and the configuration of the different test cases.

2.3. System qualification

As soon as some IEDs are available and qualified, manual partial qualification of the system can be started to validate its functional interoperability. These tests are performed on consistent subparts of the system: the HMI, telecontrol gateway and substation level automations (environment GBb in Figure 3) to validate the station level interoperability on one hand, and bay level IEDs on the other hand.

In parallel, a system qualification testbed is used to perform automatic testing. Partial integration tests are executed incrementally as components are delivered, until the entire R#SPACE system is available. The main aspects of system qualification test strategy are summarized here:

• The integration test plan must be automatically executable, to enable validation of multiple IED combinations within a limited time frame. For that purpose, tests are developed using Python, and launched by a program that parses the configuration files (SCD file, parameters files, wiring documentation…).
• The communication of IEDs that are not physically connected must be emulated by the test tool, especially when topology of substation for the PACS under test becomes bigger than the capacity of the laboratory. Real IEDs also need to be monitored so that their communication and data model are accessible to the test tool.
• It must be possible to use hard wired connections (PACS process interface) between the system under test and the test tool, to inject current and voltage (emulation of Voltage Transformer / Current Transformer), or to stimulate the logical binary inputs or outputs connected to the SCU, associated to HV equipment.

Regarding the content of the tests, a system qualification is organized in 4 steps:

• Preliminary tests are first carried out to verify that the initial conditions to perform the test are reached. These preliminary tests cover configuration (data model in IED versus SCD file), terminal configuration (conformance between signal and physical I/O), availability of test tool features, and LAN verification (synchronisation, supervision of GOOSE and SV subscriber).
• Functional tests are performed to validate the exchanges of data information between applications in a system context, including physical I/O of the system and the global time synchronization of the applications.
• IEC 61850 tests allow to verify that the standard mechanisms, such as management of the test mode and the simulation bit, or substitution, are properly implemented and propagated in the system.
• System level tests are finally performed to verify the global requirements including supervision or cybersecurity, or to implement specific system use cases based on experience acquired on other systems.

To be able to apply this test strategy, RTE has specified and developed a specific integration test tool with a supplier mainly specialized in aeronautic testbeds. Figure 5 below shows this test platform, which contains voltage and current injection capabilities, binary inputs and outputs racks, a real time server hosting IEC 61850 MMS, GOOSE and SV features, IEC 60970-5-104 center emulator, and other physical racks such as Modbus or analog sensors.

2.4. Factory Acceptance Test (FAT)

In the R#SPACE project, it has been decided to delegate assembly of components of the PACS in cubicles and kiosks and the FAT before the on-site installation to external stakeholders, called “Systemier”.

Whereas the initial plan was to use the same automatic test tool as system qualification (as described in §2.3 and Figure 3) with a specific HMI, it appeared that the tool ergonomics and the overall strategy were not adapted to an external contractor and the expectations of the FAT. The methodology ultimately adopted consists in carrying out end-to-end manual tests. The Systemier is expected to analyse and characterize the anomalies encountered. This requires good knowledge of the system. The first FAT was carried out at RTE's Transfo site, in parallel with the system qualification, with strong support from RTE teams. For future deployments, tests will be carried out on the Systemier’s premises.

2.5. Site Acceptance Test (SAT) and commissioning tests

The SAT and commissioning tests are intended to verify the various interfaces with primary equipment, auxiliary installations, other substation installations, monitoring sensors and telecommunication interfaces. For this reason, most end-to-end tests during SAT of the different functional chains are performed based on a detailed test plan defined by RTE. This ensures a complete and homogenous coverage of these tests, including those carried out by contractors.

Verifying the correct connection and configuration of the instrument transformers is within the scope of FAT and SAT. This concerns the orientation of the currents, the correct configuration of the scaling factors and the correct association of phases to the different channels in the SV flow. The use of the local HMI as support for the test has proven to be efficient. This includes the possibility of displaying the phasor diagrams for voltages and currents based on acquired SV streams.

The different features of sub-functions are tested previously (see §2.2, §2.3, §2.4) and do not need to be tested in depth during SAT [3]. Being executed at PACS process interface level for both analog inputs and wired binary inputs and outputs, the SAT and commissioning tests for R#SPACE systems are therefore in principle identical to those of conventional PACS and can be performed using similar test tools. There is one exception regarding the tests performed for feeders containing primary equipment directly interfaced to the PACS via IEC 61850. In the case of LPIT (Low Power Instrument Transformers), an analog injection on the secondary side of the Instrument Transformers is in general not possible. Different test procedures must be used in this case [4].

2.6. Post commissioning tests

During the lifetime of the PAC system, tests may be carried out in the event of malfunction or evolution in the applications implemented. In principle, there is no difference for the associated functional tests between a conventional system and one based on IEC 61850. However, the fact that wired information has been replaced by IEC 61850 digital services changes the actions to be undertaken to isolate a part of the system to be tested.

In IEC 61850 full digital systems, several PACS functions are hosted by the same IED. The IEC 61850 interfaced functional chains span over several IEDs, e.g. SAMU, protections IEDs, and SCU. It becomes thus necessary to use test mechanisms and features provided by IEC 61850. The combination of test modes (on, on-blocked, test, test-blocked, off), LPHD.Sim data object (put all the applications hosted by the IED waiting for simulation message) and the use of test tools with test mode or simulation bit enabled, makes it possible to set up isolation mechanisms at application and IED level. However, a strong analysis is required to set up the right test configuration depending on the expected test case, which presupposes expertise in the IEC standard.

Other, more refined mechanisms can be used to manage the switch to a test source at the level of an application input (with data object InRef), enabling much more targeted test procedures to be set up. But these mechanisms are less widespread today and will require an in-depth laboratory test phase before being used on an operational site.

3. Project progress

The R#PSPACE project was launched in 2017. The development is based on several phases, each of them covering a specific subset of the specifications. The first phase (phase 1) of the project concerns small substations with one voltage level and a limited number of transmission lines, mainly 63 kV to 90 kV. Phase 2 will address substations with several voltage levels of RTE, up to 400 kV with more complex busbar topologies, and all corresponding PAC functionalities.

For each phase, testing is performed in a laboratory in RTE facilities in Transfo site, based on a generic version of the system. When qualified, a generic version can then be instantiated in real substations that are compliant with this generic version (availability of all the required functions).

To gradually develop our expertise, the first qualification of the R#SPACE system was performed in laboratory in 2023 with the exact configuration of the pilot substation. Automatic testing was developed in Python, for around 60% of the applications of the system. Due to time constraints, it was decided to conserve some manually managed tests, and to spread the system qualification activity over two testbeds (system qualification and FAT) running in parallel. This adapted organization has enabled us to perform system qualification of the pilot substation in less than 6 months.

Anomalies encountered during this phase were analyzed on a daily basis by all R#SPACE project experts, in order to determine their level of priority and to assign a project lot to deal with them, so that we could achieve an acceptable level of system quality for commissioning the pilot substation. Figure 7 below gives a breakdown of the types of anomalies detected during the system qualification and FAT tests of the pilot substation.

The first pilot substation running R#SPACE system was commissioned in December 2023, six years after the beginning of the project. The next steps will focus on the qualification of one real substation configuration covering several instantiations, and at the end, the configuration of a generic substation will be delivered to validate all the requirements of phase 1.

This generic configuration will then be used as a base for the industrial deployment of R#SPACE PACS in the following years.

4. Selected issues detected during testing and associated recommendations

4.1. Testing coordination and version management

Due to different development schedules and delivery dates for each component or the correction of anomalies detected during component qualification, certain components or their updates were available at different times in the overall project timetable. That means that the sequential process of test from conformance to system qualification is often unrealistic. In this context, it becomes necessary to determine at what stage of development an IED can be integrated into the system, which non-regression tests need to be carried out when an updated version is available, how to manage the test results obtained on different testing environment, etc. A strong test management must therefore be implemented to maintain a global overview of the interoperability and qualification tests results. For the R#SPACE project, daily team meetings were held during the system qualification and FAT, as explained in §3. It was also helpful to rely on dedicated, vendor-independent tools to share and classify the anomalies and evolutions of IEDs within the project.

Another difficulty is the management of results that are not consistent between different testing environments. This raises the question of the representativeness of the different test environments, which by nature cannot be totally identical. For example, on the system qualification platform, monitoring tools are permanently installed, which modifies the traffic on the LAN by adding IEC 61850 traffic (polling in particular). The network configuration is also adapted so that test tools can be connected and have access to all data flows, regardless of the LAN segregation mechanisms put in place. A detailed knowledge of the platforms enable assessment of the probability of an anomaly being reproduced in the context of a system in operation. In order to carry out this analysis, the identification of all the components of the system under test must be traced, which is a complex task given the large amount of data that needs to be centralized (hardware/software version/configuration of the IEDs, LAN configuration, OS version of the virtualized infrastructure, versions of the IED Configuration Tools or System Configuration Tool, etc.). This management needs to be anticipated by implementing appropriate tools or databases.

4.2. Qualification of SCL files

In a top-down engineering process, the SCL configuration files, and in particular the system configuration (SCD file), are deliverables that must be integrated into the test schedule. Even before starting functional tests at system level, it must be ensured that the configuration from the System Configuration Tool can be loaded into the devices and lead to a functional configuration of the IEDs. On the R#SPACE project, it took several weeks of exchanges between the suppliers and the team in charge of configuration to achieve an operational configuration. A first recommendation is therefore to anticipate this phase of convergence around the configuration files in the planning. Another action would be for suppliers to anticipate developments with their own configuration files.

Furthermore, even after the initial validation phase of SCL files, numerous changes were made to the system configuration, either related to the correction of anomalies, or to implement changes aimed at managing some undesired system behaviors. In this context, it is necessary to have a common understanding which type of detected error or deviation should be tolerated by the different configuration tools. The question of the rate of version upgrades on the system qualification platform was raised, and it soon became clear that it was not possible to update the system under test with every SCD configuration delivery. IED qualification platforms were set up to validate each configuration delivery, while a periodic update logic was implemented on the system qualification platform, starting with an initial rate of 2 updates per month. At the same time, a prioritization process has been put in place to limit configuration changes to what is strictly necessary. Ultimately, the configuration must be definitively stabilized before the final system test phases, so that all the system tests can be carried out on the target environment.

4.3. New requirements for staff skills

Digital systems rely on an elaborate IT infrastructure, from local network management to virtualization environment. As a result, test operators need to combine traditional electrotechnical skills applied to real-time systems with IT skills and, in the case of automatic tests, programming skills.

To illustrate this point, during the test phase of the R#SPACE system some issues were encountered regarding the LAN configuration. After analysis, it appeared that the maximum Ethernet packet size on switches, IEDs and servers had to be adjusted to ensure that all dataflows were transmitted correctly. Another anomaly identified during the system redundancy tests was linked to the lack of compatibility of the PRP card used in the virtualization servers. Resolving these anomalies required considerable expertise in communication networks and IT infrastructures. This activity was further complicated by the absence of a specific contract for the hardware used for the virtualization servers in the R#SPACE system, and therefore the absence of contractual support from the supplier.

Digitalization is also opening up new challenges in terms of cyber security. In order to improve the reliability and availability of PAC systems, mechanisms based on role-based access control, firewall management or dataflow encryption are being put in place. It appears necessary to think about how activating some of these features in the integration test phase. Testing a system in a cyber-secure environment sometimes means adapting the standard procedures. In order to not reduce the effectiveness of the test phases, it is advisable to include test use cases right from the system design stage. This requires strong cyber security skills to be built in from the outset.

Multi-disciplinary teams can help solve this challenge of a broad skills scope, and maintain sufficient efficiency with test systems and analysis tools that are increasingly digital and converging between OT and IT.

4.4. Detection of hidden issues

System qualification for the R#SPACE pilot substation was carried out by testing each application within the whole system. All requirements related to exchange between applications were verified by unitary tests that were replayed several times and in particular each time when the system is upgraded or modified. This allowed us to detect functional issues, mainly related to IED capability or configuration. In the meantime, testing tools and the local HMI of the R#SPACE system were used to continuously monitor the events in the system qualification platform, even after the official system qualification period has ended. These tools enabled us to detect events in the dataflow (sequence or count jumps between GOOSE or Sample Values messages), loss of synchronization, unexpected changes in digital input / output state, etc.

During R#SPACE system qualification, although all the application requirements had been checked, intermittent malfunctions were detected in the system every few days. For instance, some IED stopped sending GOOSE repetitions messages for few seconds to hours. This anomaly has not been detected during qualification test of the IED, and the apparently random conditions of occurrence made the reproduction in the supplier’s facility difficult. Three months had been required to understand the factors triggering this anomaly.

That means that system qualification requires both specific tools to monitor all the events in the system, and appropriate time to be confident that the system will perform as expected over time. Continuous tests and stress tests must be planned as early as possible, from IED qualification to system qualification in the stabilized environment, to be in capacity to detect and analyze the non-functional anomalies and solve them in time.

5. Conclusion

With the R#SPACE project, RTE has designed a multi-vendor PAC system. The first pilot site was commissioned at the end of 2023. This pilot site has allowed to validate the entire test approach, from equipment development to qualification of the whole system. One of the key lessons learned from this first stage is that testing a multi-supplier system is no more complex or time-consuming than testing a turnkey system. Most of the anomalies encountered are not related to the interfaces between devices, but rather to non-conformities at the level of a particular IED or configuration files.

It has also proved extremely advantageous to control the entire test chain. This makes it possible to prioritize the tests according to the delivery schedules of the various project teams or the expected verification points in a very flexible way, to validate the patches iteratively and progressively, and to carry out tests over long periods (several months). During the maintenance phase, the availability of test tools should enable us to respond quickly to problems encountered by the operational teams. The development of automatic tests based on the mastery of the test tools and the analysis of results from code programs (such as Python) is a leverage for making the test phase more efficient, notably through the reusability of tests and the speed of execution of finalized tests.

Now that both the PACS design and the test approach have been executed and proven, the challenge is to successfully move from qualification testing for a pilot site to qualification testing for a generic system covering a whole range of functionalities; and to successfully upscale this testing activity. The right tools, processes and skills need to be put in place and maintained over time to ensure that the testing phase remains industrial and efficient. Arrangements for sharing testing tasks with external partners need to be studied in order to find the best compromise for retaining the benefits of bringing part of the test chain in-house, while limiting the material and human investment required internally. This approach implemented in the R#SPACE represents major investment for a TSO like RTE. In view of our initial results, we are now confident that the expectations for the project can be reached.

References

1. RTE Substation Protection Automation and Control Systems IEC 61850 Model ind 7 https://www.rte-france.com/fournisseurs/network-together#LeprojetRSPACE
2. Test and Integration approach for RTE’s R#SPACE Protection Automation and Control System – PW47 PAC Word Conference June 2022
3. CIGRE WG B5.53: TB 760 "Test Strategy for Protection, Automation and Control (PAC) Functions in a Fully Digital Substation Based on IEC 61850 Applications" March 2019
4. J-Y. Astic, V. Leitloff, T. Buhagiar, A. Kurtz, P. Brun, S. Vigouroux, J-P. Cayuela: "'Postes Intelligents' Design, Test and Commissioning of RTE's first completely Digital Substation" PAC World Magazine June 2018
5. T. Michel, V. Leitloff, JM. Boisset, X. Michaut, J. Brengarth, S. Germain: “R#SPACE – RTE’s new approach towards fully digital Protection Automation and Control Systems” Paper PW034 PAC World Conference September 2021

---

• Contact Author: M. MERLEY