rsc's Diary: ELC-E 2022 - Day 3

The Convention Centre is directly at the water front of the Liffey river, just a few walking minutes away from O'Connell Bridge, Temple Bar and Trinity College. Visiting ELC-E is always a good opportunity to visit interesting cities in Europe. However, here is my story of the talks I heard on day 3.

Continuous Delivery of IoT Sensors

The first talk on Thursday, by Ulf Lilleengen (Red Hat) reported about the Drogue IoT project, a system (written in Rust) for continuous IoT deployment.

The setup consists of:

  • continuous integration
  • continuous testing (for any change)
  • continuous deployment (when a step is reached that makes sense to be deployed)

The devices he cares about are not in the Linux league, they are microcontrollers that have an updating mechanism built into the application. The devices are connected in a different, in most cases wireless (WiFi, LoRaWAN, LTE-M, Bluetooth), way, and usually use different sets of tooling for the actual firmware build (vendor toolchains). Some of the channels might be air time restricted: LoRaWAN for example might need three days to update a 64 kB firmware image when using a standard gateway. To make the devices failsafe, they need at least a minimal bootloader that is able to fetch an update image and persist it on the device.

Communication in Drogue IoT works via several API endpoints with a REST API. Devices can be grouped into applications, and many different protocols can be used to talk between the backends and devices. Basically, the devices ask if there is a firmware update available for them, and the delivery service answers. The actual delivery can for example be done with HawkBit. Finally, a demo showed how to roll out firmware for a MicroBit controller.

Many of the components don't really fit our Embedded Linux environments, but there might be some interesting components to have a closer look at, such as keycloak (for identity management) or the used communication protocols.

Beyond Complex Cameras: Complex Video Graphs Using PipeWire

George Kiagiadakis (Collabora) then motivated recent issues with increasingly complex cameras. Modern cameras might have several sensor chips, combination algorithms and lots of hardware units, all resulting in one picture or video, maybe even with AI accelerators being involved. However, the processing pipeline to manage all of that becomes increasingly complex. Some of the functionality for this is nowadays provided by libcamera, but there are unsolved questions:

  • How to combine data from separate devices?
  • Can we separate processing, even into several containers?
  • How to realize a "divide and conquer" strategy?

PipeWire was invented as a multi process, resource sharing, low latency multimedia bus. A processing graph, split across several processes, is constructed and can be managed by WirePlumber. WirePlumber is a scriptable daemon and can discover and connect all devices in the system.

As a demo, George showed a PipeWire demo with a face detection algorithm. There is a nice little tool that shows the resulting PipeWire graphs. A 2nd demo showed a video playback usecase with two output windows connected to one video decoder, and the setups can even be combined.

The talk gave a nice overview of PipeWire; our experience is that it appears in more and more recent Linux media projects.

Growing a Lab for Automated Upstream Testing: Challenges and Lessons Learned

In my next talk, Laura Nao (Collabora) reported about the struggles from Collabora's automated test lab. They are operating a lab for automated testing with LAVA in Cambridge; the lab consists of 158 devices of 32 different types, controlled by 15 servers and the usual set of networking switches, debugging interfaces, USB hubs etc. Most of their devices are ARM64 chromebooks, often more than one device of the same sort. They have the boards in 19" racks. Sounds pretty similar to what we have in the Pengutronix LAVA lab, only that the Pengutronix lab has more industrial style hardware.

The test flow starts with switching on the device, running the bootloader, stop booting, then transfer the kernel and ramdisk, setup the commandline, resume boot, then start the kernel, boot, login, then run a test script and finally turn the board off again. The device configuration consists of a set of jinja 2 and yaml files. A health job with a known-good result checks if the lab works correctly. Laura showed the different test job configurations. A dmesg job checks the kernel log for errors. A LAVA GitLab runner makes it possible to schedule jobs in GitLab, then run them on the LAVA lab. For new devices, they run 1000 health checks before sending the device into production.

The actual test flow for chromebooks looks pretty different from normal Linux applications. As testing on Chrome OS is difficult, they replace it by a normal Linux.

The lab is split into a staging and a production area: in staging, lots of health checks are provided. It is also the place where the upstream is tested. Staging usually runs the latest and greatest. The production area of the lab continuously supervises released revisions.

The lab is fed by KernelCI and Mesa's CI and helps the upstream developers both inside and outside of Collabora to spot bugs as early as possible. Mesa is relatively new and does pre-merge conformance testing; the majority of tests run on x86.

As there are many jobs in the lab, a lot of things may go wrong: sometimes the hardware is flaky, sometimes the servers, sometimes the configuration in the racks do fail and even firmware bugs might kick in. As Mesa blocks new merge requests when the tests fail, it is important to differentiate infrastructure fails (i.e. network failures) from real test fails.

Their findings are:

  • Monitoring the devices' health status is important, as are robust health checks.
  • LAVA infrastructure errors need to be found.
  • With enough device redundancy, good device coverage is possible.
  • It is necessary to have two people on-side to maintain the devices.

With the growth of the lab, more space, equipment and maintenance manpower is needed as well as more database performance. Monitoring that closely, it was possible to optimize LAVA for better performance.

Technical Introduction to EVerest: Open Source Firmware for EV Charging Stations

Renewable energy is currently an interesting topic for us, and once we'll have infrastructure at Pengutronix, we'll of course run it with as many open source components as possible. So having a look at what Kai-Uwe Hermann and Piet Gömpel (PIONIX GmbH) have to tell sounds interesting.

EVerest is a software stack for charging infrastructure for vehicles. It turns out that charging is a surprisingly complex topic. Charging stations usually consist of a power component and a control component, the latter one often running embedded Linux. One has to deal with lots of different standards, both on the connector and communication side. Then, between the charger and the cloud backend, the OCPP protocol is used (implemented in more than 150 dialects), but also IEC631100, MQTT and others. There is an ongoing effort to establish a "plug and charge" mechanism, but it requires complex authentication, so almost nobody uses that in reality.

Another topic that needs to be cared of is to interact with local charging infrastructure, i.e. for controlling to charge when the sun is shining and a local PV system is producing energy. In this area, there are even more protocols involved. The situation is similar when it comes to grid communication.

All in all, there are way too many standards, and it's even growing. As this cannot be solved by adding another standard, the EVerest tries to provide a community reference implementation. They try to support as many hardware platforms as possible, while not reinventing the wheel. The Apache 2.0 licensed code base is written in C++ and can be found on GitHub. To make the configuration easy, there is some web application. Currently they support OCPP 1.6 in a feature complete way and aim for 2.0 support. In addition, they have a little flutter app for the GUI on the local charge point.

The project is about two years old, has a mailing list and regular meetings. It runs on some commercial hardware (Phytec, Mahle), but nothing ready-for-the-market yet.

A Month Off: Migrating a Robot Controller from the Proprietary INtime RTOS to Linux

Dirk Eibach works at Carl Cloos Schweisstechnik GmbH, an industrial welding robot manufacturer. In 2021 they started to migrate a robot to Preempt RT.

The robot controller consists of an inhouse developed PC, with CAN bus, EtherCAT and a bunch of servos that run as EtherCAT slaves. The axis run in a cyclic synchronous position mode (CSP): On each 1-4 ms bus cycle, the system provides a new target position for all axis. The motion planner runs at 8...16 ms, interpolating the faster bus cycle. The dynamic is simulated with a physical model of the robot.

The code base is about 3 million lines of source code, originally written in PL/M, later converted to C; it still contains 386 code and ANSI C, an old MFC GUI and a new Qt GUI (still not feature complete). There is the usual amount of technical debt in the codebase, including knowledge from people that have left the company.

In 2021 they talked to KEBA, they had an automation controller (D3) running Linux (Debian 32 bit) and Preempt RT, with an integrated safety solution. With all software running on the system itself, they could get rid of lots of proprietary licenses. In this situation he proposed to try a 6-week porting sprint and try to port everything to Linux; surprisingly, everyone agreed. One of the challenges was to create a comfortable environment for all of the (non-Linux) developers.

The whole API was completely written in userspace, as a shared library, so the whole application wasn't touched at all. After some header file hackery, all functions the linker missed had to be implemented. At the same time the GUI developers started to migrate the GUI. However, some more horrifying features like doing IPC on physical memory resulted in some application redesign.

After 6 weeks, the Cloos main realtime applications were starting up; the GUIs started working, but were not finished. They learned a lot from the effort and it was definitely the right approach: the team gained a good feeling about how complicated it might be to do the rest of the port. Using Visual Studio made the former windows developers feel comfortable, so there was a collective decision to continue the effort. Unfortunately, then the chip crisis hit the team and they had to focus on other tasks first.

Further Readings

rsc's Diary: ELC-E 2022 - Day 1

After two years with online conferences, the embedded Linux community once again meets in person for their annual Embedded Linux Conference Europe in Dublin, Ireland this year. Since many years, ELC-E is part of the Linux Foundation's Open Source Summit, and the best opportunity of the year to meet other Linux kernel and wider core ecosystem developers and discuss recent and future technological topics.

rsc's Diary: ELC-E 2022 - Day 2

The Dublin Convention Centre is huge - there is more than enough space for all the developers participating in the Open Source Summit. Fortunately, the talks will be on YouTube after the conference, so it's no problem that one can only hear a small selection of talks. However, here is my report of the talks I heard on the 2nd day of the conference.

rsc's Diary: ELC-E 2022 - Day 4

Friday, the last day of ELC-E 2022, is traditionally the day of the Embedded Linux Closing Game, with Tim Bird reporting about the embedded Linux world (universe?) domination progress and the overall state of the union. Of course, there were again several interesting talks.

Pengutronix at Embedded World 2024

Meet Pengutronix at the Embedded World 2024 in Nurnberg! You find us, as always, in hall 4, booth 4-261. As usual, we will be showing demonstrators on current topics at our exhibition stand.

Chemnitzer Linux-Tage 2024

Pengutronix war auch in diesem Jahr wieder auf den Chemnitzer Linux Tagen dabei. Wie jedes Jahr sind die CLT eine willkommene Gelegenheit Freunde zu treffen und sich über Linux, Open Source und den Rest der Welt auszutauschen.

FrOSCon 2023

In a few hours, the 18th FrOSCon will begin at the Bonn-Rhein-Sieg University of Applied Sciences. Pengutronix will be there again with a small team. At one of the partner booths we will show some of our activities in the open source community. We will bring our labgrid demonstrator and the FPGA demo.

Embedded Linux Conference Europe 2023: Our Recommendations

Last month Pengutronix was present at the Embedded Open Source Summit (EOSS) in Prague. Thanks to all to all speakers for sharing your knowledge! In this blog post we want to shine a spotlight at a few talks that we found especially interesting. (Links to recordings will be added once the recordings are available.)

Embedded Linux Conference Europe 2023: Our Contributions

This year the Embedded Linux Conference Europe (ELCE) is back in Prague! Pengutronix, again, is on a field trip with 15 colleges to attend the conference. The ELCE is one of the big conferences where the Embedded Linux Community meets during the year. This time the ELCE is part of the Embedded Open Source Summit (EOSS): a new conference with only embedded topics and without cloud- or crypto-tracks.

Pengutronix on ELC 2021

The sun is shining for the last days of summer 2021. It doesn't only mean that autumn is coming, but also that this year's ELC is in preparation.

ELCE 2020 - Recommended Talks

The Embedded Linux Conference Europe (ELCE) is the one biggest meetup of Embedded Linux developers in Europe. As usual Pengutronix has attended this conference - but this year from the warmth of our homes.