Showcase: Preempt RT and Time Sensitive Networking

J. R. R. Tolkien already knew...

"Real Time is never late, nor is it early, it arrives precisely when it means to."

Nowadays, even small and cheap microcontrollers offer enough calculation power to perform time critical tasks within an industrial environment. However, as soon as actors and sensors are spread over an entire facility and are to be connected over Ethernet, the actual moment when a data packet will get processed becomes very hard to predict. At this point, Linux running a Preempt RT Kernel altogether with a network featuring Time Sensitive Networking (TSN) capabilities can help.

When sending data it is likely that delays not only occur inside the processing system, but also on the transmission path. As such, critical data from multiple sources might arrive at the recipient at once. The more modern embedded and connected devices allow for fast calculations and data handling, the more the question arises not only if your system performs "fast enough", but whether your results also arrive "in time".

The usage of complex network stacks based on standard Ethernet may be the motivation to swap the microcontroller with a yet more capable soulution based on a microprocessor running a proper Linux as operating system. Here, Real Time features are implemented by the Preempt RT mechanism of the Linux Kernel, and time critical data transfer also gets covered by more and more TSN features being added over the time.

TSN Demonstrator

Being a very heterogenous topic, TSN for itself consists of a variety of technologies: On one hand, not all standardized processes are already available under Linux, while on the other hand not every implementation even requires the full set of available features.

In order to gather further insights into the technical challenges that occur during the development process we at Pengutronix have designed a TSN Demonstrator.

The "Floating Sphere" is a popular experiment where the basic concept of a PID control loop is demonstrated. In that setup a magnet is attached to a lightweight object (e.g. a paper ring). This shape is then made levitating in the field of an electromagnet: A sensor detects the position of the ball and reports it to a controller, which in return regulates the current through the electromagnet accordingly to a constant distance from the sensor - when done right, the sphere keeps floating in an equilibrium as long as the setup is powered.

To simulate a system that is connected via TSN Ethernet the sensor, actor and controller were realized on single-board computers that are based on the STM32MP1 microprocessor running the latest Linux Kernel (5.11) and Preempt RT. The boards are connected over Ethernet and Precision Time Protocol is used to synchronize the clocks of the participants.

Goals

The Demonstrator setup inherits various latencies when the process of regulating the paper ring's position takes place:

  • The information gathered by the sensor has to be processed by the first system.
  • The packet has then to find its way through the network and to the board with the controller software.
  • The controller must calculate the electromagnet's next state whilst sharing the board resources with other programs and services.
  • The resulting values again are to be sent over the network to finally reach the actor's system to get applied at the right moment.

With the system as a whole now being to slow to maintain a stable floating position the individual components have now to be coordinated in real time.

Subsequently, we realized our PID-loop as follows:

  • By using PTP (IEEE 802.1AS), the internal clocks of our boards were synchronized within in the margin of less than a microsecond.
  • Critical latencies within the software environment were tracked down and solved if necessary.
  • The positioning sensor is being read synchronous to the PTP timer and a respective time stamp gets applied; the daten then needs about 600 µs from sensor to actor.
  • The PID controller runs at a frequency of 1.25 kHz / 800 µs.
  • Lastly, the actor's software has to take care that the new sVtate of the electromagnet is applied at the right time.

The result can be seen in the accompanying pictures.

Next steps

The Demonstrator setup allows for first insights into the different technologies around TSN under Linux. During the development a range of current SoCs was evaluated regarding their TSN capabilities: Not all features are available at every hardware, so the SoC has to be chosen specific to the task at hand. Additional room for improvments also exists on the software-side - many of the used technologies are still young and quickly evolve from Kernel to Kernel. It is therefore recommended to always use a current version of Linux.

How can Pengutronix help with TSN projects?

More about TSN and PTP

Our friends at OSADL are constantly conducting measuring tasks regarding the performance of Real-Time and TSN systems. If you are interested, you may check out their article "Real-time Ethernet (TSN) synchronization analysis". (osadl.org)

The Pengutronix Kernel-Team can advise industrial customers on the development of TSN systems as well as directly support them with development services. An important task is the further development and adaption of the Linux Kernel to the individual requirements of the customer's project, for example through new drivers for switch chips and TSN features in network hardware.

Back to technical showcases

Further Readings

Showcase: Embedded off-the-shelf

A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.


Showcase: Continuous Testing

About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.


Showcase: Remote Working

Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.


umpf - Git on a New Level

Modern software development is commonly accompanied by a version control system like Git in order to have changes to the source code being documented in a traceable manner. Furthermore, any version state should be easily reproducible at any time. However, for work on complex projects such as the BSP ("Board Support Package") of an embedded system with its multiple development aspects, the simple stacking of the individual changes on top of each other does not scale.


Pulse Width Modulation (PWM) is easy, isn't it? - Turning it off and on again

Part of Uwe Kleine-König's work at Pengutronix is to review PWM (Pulse Width Modulation) drivers. In addition, he also sometimes refactors existing drivers and the Linux kernel PWM subsystem in general.


Yes we CAN... add new features

Have you ever experienced an otherwise fine product that is missing just the one feature you need for your application?


DSA in Barebox

The v2022.05.0 Release of barebox introduced initial support for the Distributed Switch Architecture (DSA) Framework. DSA is originally a subsystem from the Linux Kernel, which exposes the individual ports of a network switch IC as virtual network interfaces.


Pengutronix at Embedded World 2022

Welcome to our booth at the Embedded World 2022 in Nürnberg!


First Steps using the candleLight

So you went and got yourself one of our fancy rocket-penguin branded CandleLight dongles or, being the die hard hacker you are, went and soldered one up in your toaster oven labeled "not food safe". What's next then? How do you use this thing? Let's answer these question by grabbing a Raspberry Pi and exploring some of the possibilities.