rsc's Diary: ELC-E 2022 - Day 2
The Dublin Convention Centre is huge - there is more than enough space for all the developers participating in the Open Source Summit. Fortunately, the talks will be on YouTube after the conference, so it's no problem that one can only hear a small selection of talks. However, here is my report of the talks I heard on the 2nd day of the conference.
Board Farms for Everyone: Making Hardware Debugging Easier and Sharing Boards Across the Globe!
Open Source is about collaboration (often even between competitors working in the same market segment), and thus this year's labgrid test lab automation talk was not given by Pengutronix, but by Christopher Obbard (Collabora).
In the beginning, Chris motivated that it is often necessary to "try something out" on a board, but this board could be in some remote location. So flashing a remote board and automated testing is something the Collabora guys needed as much as we did. Chris explained that, in his local lab, he needs about 10 boards, the lab needs to be reconfigurable quickly, it has to be low power and silent (WAF!). His lab uses a Raspberry Pi 4 as a controller with USB serial converters for the console (with the same robustness issues we experienced) and a network switch. He uses ansible to set up the software on the Raspi.
On the software side, he started with LAVA, but quickly found that it is mainly made for automated test runs, but not for interactive work. He found the maintenance effort to be high; flashing full disk images is difficult, and the loop to run a job takes quite a long time. So for a home lab and interactive work with a focus on actual development, LAVA was not suitable.
For interactive work, controlling the power and turning boards on and off is important, as well as flashing and booting into the software. The same requirements lead us at Pengutronix to developing labgrid, as a python framework for hardware test automation. A controller maintains system state and is able to connect to several exporters that actually talk to the devices, abstracting things like power ports and serial interfaces. Chris found that the simplicity to configure labgrid is really a bonus: just write two YAML files with a hardware description and places and be done. For interactive use, you can lock and unlock a board, so it is possible to share a place with a CI. To switch 230 V devices, he uses Sonoff WiFi PDUs that can be controlled with MQTT. For automatic tests, Chris uses GitLab CI.
All in all, Chris seems to be quite happy with his lab. A small pain point is that there is no standard way to upload files to the exporter. However, it is good to know that people seem to be happy with our open source projects...
Using OpenEmbedded Inside Containers? How and Why?
The 2nd talk I heard was given by Drew Moseley from Toradex, who reported about his experiences with OpenEmbedded inside containers. According to his experience, containers are a convenient way to package software without dependencies to the host system and deliver it to the embedded system. Containers mainly make use of standard kernel functionality to encapsulate software; however, in contrast to virtual machines, there is no separate kernel. Drew outlined that, with container registries like Docker Hub, it is pretty easy to install any kind of software without struggles.
The main question is: how do you create containers? One common problem with predefined container registries is that you have to trust a container image made by someone else, and the usual distribution containers include a lot of standard software that might not even be needed. With OE, pretty small container images can be built - I leave the details out here, so if you are interested, the video of the talk should be available soon.
Leveraging Next Generation Cellular Networks for Drone Telemetry and Payload Communication
Unfortunately, the "How Automotive Grade Linux UCB Transitioned to Make Use of Flutter" talk was cancelled; the next talk by Ngonidzashe Mombeshora couldn't be held in person due to visa issues, but was shown as a prerecorded video.
The advantage of using mobile networks for drone control is that cellular networks are available almost everywhere. However, to be able to fly a drone in realtime, latency is pretty important, so in order to optimize for latency, he first built an SDR based open source 4G network for drone communication, based on an SDR hat for a Raspberry Pi and a set of docker containers for the software stack. Then, a 2nd generation testbed with 5G was set up and he achieved 90/30 MBit/s down-/uplink at 5 ms latency. Compared to commercial networks, the testbed had much better performance w.r.t. latency. Finally, he run a drone testflight in his own non-public mobile network, using MavLink for telemetry.
All in all it was an impressive talk, but of course mainly from an academic point of view.
Automating and Managing an IoT Fleet Using Git
In my last talk of the day, Matthias Lüscher (Schindler AG) talked about how to manage an IoT fleet with git. The mission of his open source project, edi, is to automate as much as possible for building, managing and delivering embedded Linux for embedded devices. The setup starts with Debian for the packages, he then constructs a Mender update image and deploys it to a test field first, runs his testsuite with Testinfra and gets test results back. If everything goes well, he gets a qualified image for the target device.
To avoid a golden image approach, Matthias uses debootstrap, runs the resulting minimal Debian image in QEMU and integrates the necessary changes and configs with the help of ansible. Testinfra turned out as a convenient environment to write tests in pytest, as it can be run on a development machine and work against an embedded target.
As an example, Matthias showed an edi workflow that turned a Raspberry Pi into a GitHub Action Runner, using the tools outlined above. In result, the runner registers itself at GitHub. In a 2nd example, a Raspberry Pi was turned into a Kiosk computer with 6 lines of ansible code.
In order to move from a single device to a whole fleet, Matthias showed his home demo fleet with devices from legacy Raspberry Pi 2 up to current i.MX8M evalkits. He models his fleet in a set of git branches: a development branch is where all of the development takes place. Then a main branch collects more complete features, and a canary branch is what is delivered to for example a realistic but nicely accessible environment. If nothing explodes there, things are rolled out into the real production environment. Clients usually work in pull mode, he uses Mender and the clients look if there is something new for them to pull in.
As devices might be offline at the moment of change, it is necessary to monitor the field setup, and for huge fleets, device definitions will probably not be put in an ansible inventory but into some database. However, small fleets work nicely with everything just living in git.
The toolbox turned out to be extremely powerful, and it is even possible to exchange the individual components, i.e. use Yocto instead of Debian etc.
Booth Crawl
Traditionally, there is a booth crawl where several projects show their embedded Linux demos. Unfortunately, this year there were only 6 demos, presented by three groups, one of which was Pengutronix. Hopefully there will be more in the next years, as the demos are always a highlight.
The Pengutronix crew demonstrated the labgrid test automation framework, the LXA Test Automation Controller (TAC) prototype, the RAUC update framework and our recent mainlining activities for Rockchip SoCs. Like every year, there have been lots of interesting talks with an interested audience.
Further Readings
rsc's Diary: ELC-E 2022 - Day 1
After two years with online conferences, the embedded Linux community once again meets in person for their annual Embedded Linux Conference Europe in Dublin, Ireland this year. Since many years, ELC-E is part of the Linux Foundation's Open Source Summit, and the best opportunity of the year to meet other Linux kernel and wider core ecosystem developers and discuss recent and future technological topics.
rsc's Diary: ELC-E 2022 - Day 2
The Dublin Convention Centre is huge - there is more than enough space for all the developers participating in the Open Source Summit. Fortunately, the talks will be on YouTube after the conference, so it's no problem that one can only hear a small selection of talks. However, here is my report of the talks I heard on the 2nd day of the conference.
rsc's Diary: ELC-E 2022 - Day 3
The Convention Centre is directly at the water front of the Liffey river, just a few walking minutes away from O'Connell Bridge, Temple Bar and Trinity College. Visiting ELC-E is always a good opportunity to visit interesting cities in Europe. However, here is my story of the talks I heard on day 3.
More Conferences in September: Yocto Project Developer Day and KiCon Europe
September 2024 brings a wide variety of conferences: Pengutronix will present talks at the ELCE, Linux Plumbers Conference and All Systems Go. Additionally we will attend two more conferences: The Yocto Project Developer Day in Vienna and the KiCon Europe in Bochum.
Pengutronix at All Systems Go!
This years All Systems Go! will take place on 25. and 26.09.2024 in Berlin. The ASG is a conference about low-level user-space topics. We are happy to contribute a talk about updating systems using RAUC and composefs:
Pengutronix at the Linux Plumbers Conference
The Linux Plumbers Conference 2024 will take place in Vienna from 18. to 20.09.2024. Luckily this does not overlap with the ELCE. Pengutronix will attend the LPC with six colleagues - so watch out for our T-shirts and hoodies and and feel free to chat with us.
Pengutronix at Open Source Summit Europe and Embedded Linux Conference Europe
The Embedded Linux Conference Europe is Pengutronix' most important conference of the year. It is a good place to meet new faces in the embedded community, discuss current topics and future developments with maintainers and developers - and of course: have a beer with old friends. As usual, many Pengutronix colleagues will attend: this year we'll have a 14 person team on site. So watch out for our T-shirts and hoodies and and feel free to chat with us.
Pengutronix at FrOSCon 2024
On August 17th and 18th, 2024, it's that time again: FrOSCon will take place at the Bonn-Rhein-Sieg University of Applied Sciences in Sankt Augustin - and Pengutronix will be there again as a Partner.
Embedded Linux Conference Europe 2023: Our Recommendations
Last month Pengutronix was present at the Embedded Open Source Summit (EOSS) in Prague. Thanks to all to all speakers for sharing your knowledge! In this blog post we want to shine a spotlight at a few talks that we found especially interesting. (Links to recordings will be added once the recordings are available.)
Embedded Linux Conference Europe 2023: Our Contributions
This year the Embedded Linux Conference Europe (ELCE) is back in Prague! Pengutronix, again, is on a field trip with 15 colleges to attend the conference. The ELCE is one of the big conferences where the Embedded Linux Community meets during the year. This time the ELCE is part of the Embedded Open Source Summit (EOSS): a new conference with only embedded topics and without cloud- or crypto-tracks.