#FlattenTheCurve – Introducing Our Remote Setup

The Corona crisis is a challenge that has hit many people as well as most companies quite unexpectedly. The entire team of Pengutronix wants to thank those that currently ensure our essential supplies, health system and civil infrastructure!

But, there will also be a time after the crisis and people and economy will recover. Thus stopping our work was never an option, but we need to ensure that we operate in responsibility for our employees and our fellow people.

Two weeks ago, we introduced our strategy for protecting our staff, doing our part to flatten the COVID-19 curve and slow down the pandemic by switching to remote work.

Despite already having a few full-time remote workers at our company, most of our employees work at the main office. In normal times we are used to meeting at our coffee bar to chat, discuss recent topics and share ideas. So, being isolated is a new situation to most of us, too. However, extending our distributed infrastructure to serve all colleagues (and not only a handful remote workers) was managed in half a day and all colleagues were able to seamlessly continue working.

Luckily, we are able to automate most of the things that need to happen for embedded development, but for the few tasks that cannot be done remotely (like receiving parcels, switching cables on local hardware, etc.), there is always one colleague present at our local office.

Due to our normal workflow most of the required infrastructure was already in place, but we also had to extend it here and there to face the additional challenges of being all remote now.

We like to share recent experiences with our remote environment, that helps us to keep physical distance without social distancing:

Working Remotely on Software

For doing our everyday work, we have an extended server infrastructure to manage all our developing projects in Git repositories. Compiling code and building Board Support Packages needs lots of CPU power, so we have our ICECC compiling cluster and remote server access via SSH. We have been using this infrastructure for decades, but in this case it helped our colleagues to set up their working places at home in little to no time and continue their work as usual.

Remote Conversation

Chatting like always – IRC

IRC and e-mail still are our main communication tools.

In times of all kinds of new instant messengers flying around, IRC seems to be a bit dusty – you might think. But it perfectly fits our communication use case, with the advantage of minimal disturbances: You are not cluttered with unrelated pictures and this “messenger” is really running on our own server infrastructure. Also, most of our developers already follow various OSS projects' IRC channels, so IRC puts itself forward for this job.

Our communication via mailing lists is still unrestricted and working as usual.

Still Talking to Each Other – Mumble

While we have been using Mumble, an open source VoIP chat server, for a long time for talking to our remote-working colleagues, this voice chat has now become a coffee bar replacement. It also allows us to continue our biweekly stand-up meeting, as well as discussing topics in small groups by joining a conference channel.

The most important advice for your personal setup - no matter which VoIP-tool you are using: Configure your setting to usage of PushToTalk. Your colleagues will thank you.

Show and Tell – Jitsi

Sometimes talking to each other is not enough and it's actually helpful to share a screen, look at a piece of hardware together or just talk face to face. For these cases our admins set up a Jitsi server for video conferencing in the browser. From first experiences it works quite well, even in long-lasting conferences with a handful of clients and on a small bandwidth. We are planning to do some performance test in the next few days.

The fact that Jitsi doesn't need any clients suits our “no corporate desktop” policy well.

Working Remotely on Hardware

The main difference between Pengutronix and most other software companies is that we are working directly on hardware (e.g. for developing and testing kernel drivers). This means that our colleagues need access to custom boards.

What we can benefit from during this crisis is that fully working remotely actually is not that much different from what we usually do in our daily business already. After initial bring up, most of the customers' hardware we work on is installed in one of our several remote testing and development labs. This way, our developers can work with the hardware from their office desk and are able to automatically switch power, get a serial console, have network access, etc.

Our Remote Setup

We've worked a lot on both hardware and software tools that enable working remotely on hardware during the past years; and we'd like to give a short insight into our setup in order to possibly also inspire others:

Each remote lab consists of the following core components:

  • a network controllable (Gude or netio) power switch
  • a network controllable serial server (accessible over the RFC 2217 protocol)
  • a network switch
  • a small server

Some of them are also equipped with

  • CAN infrastructure for IO switching
  • USB hubs for controlling USB devices

Our open source hardware automation tool labgrid then brings these physical resources to the developer's desk.

Each remote lab then runs a labgrid exporter instance that automatically provides labgrid “resources” for the available remote hardware. A resource can be for example a power switch, a serial console, a USB device, or a CAN remote IO.

These resources can be grouped to places, which allows grouping resources with a similar purpose, and controlling them by a simple abstract place name like “my-devboard”.

Then each developer has a local instance of the labgrid client running that allows him to lock places, switch power, get a serial console, etc.

Finally, a central labgrid coordinator instance runs in a VM, coordinating access between the labgrid exporters and the clients.

This setup works both on local office desks as well as from home connected through SSH.

Helpful Tools

But, there are still things left that one can find hard to automate, like populating SD cards with a disk image and plugging it back into the device. For this purpose, our partner company Linux Automation GmbH brought up the USB-SD-Mux. This device consists of a small PCB with one end formed like a µSD card that you can put into your board's µSD slot instead of the physical µSD card. The physical µSD card is then put into the USB-SD-Mux instead. Connected over USB to a host computer it then allows us to switch the SD card between the host and the device under test, and re flash the SD card from the host like a standard USB mass storage.

We also have similar hardware in the pipeline or in the final test phase for Ethernet switching (like actually plugging cables in and out) and for USB switching (e.g. to simulate plugging of an USB stick). Stay tuned to find out about those :-)

We hope, you'll find our experiences helpful. Even though our remote setup works very well, we are looking forward to an end of this crisis and meeting each other in person again.


Further Readings

umpf - Git on a New Level

Modern software development is commonly accompanied by a version control system like Git in order to have changes to the source code being documented in a traceable manner. Furthermore, any version state should be easily reproducible at any time. However, for work on complex projects such as the BSP ("Board Support Package") of an embedded system with its multiple development aspects, the simple stacking of the individual changes on top of each other does not scale.


Yes we CAN... add new features

Have you ever experienced an otherwise fine product that is missing just the one feature you need for your application?


DSA in Barebox

The v2022.05.0 Release of barebox introduced initial support for the Distributed Switch Architecture (DSA) Framework. DSA is originally a subsystem from the Linux Kernel, which exposes the individual ports of a network switch IC as virtual network interfaces.


Pengutronix at Embedded World 2022

Welcome to our booth at the Embedded World 2022 in Nürnberg!


First Steps using the candleLight

So you went and got yourself one of our fancy rocket-penguin branded CandleLight dongles or, being the die hard hacker you are, went and soldered one up in your toaster oven labeled "not food safe". What's next then? How do you use this thing? Let's answer these question by grabbing a Raspberry Pi and exploring some of the possibilities.


Wir haben doch etwas zu verbergen: Schlüssel mit OP-TEE verschlüsseln

Moderne Linux Systeme müssen häufig zwecks Authentifizierung bei einer Cloud- basierten Infrastruktur oder einer On-Premise Maschine eigene kryptografische Schlüssel speichern. Statt der Verwendung eines Trusted Platform Modules (TPM), bieten moderne ARM Prozessoren die TrustZone-Technologie an, auf deren Basis ebenfalls Schlüssel oder andere Geheimnisse gespeichert werden können. Dieses Paper zeigt die Nutzung des Open Portable Trusted Execution Environments (OP- TEE) mit der Standardkonformen PKCS#11 Schnittstellen und i.MX6 Prozessoren von NXP als Beispiel.


Fixing ICECC, our Workhorse for Building BSPs

While developing operating system infrastructure for industrial devices for our customers, we build lots of embedded Linux board support packages, kernels, bootloaders etc. at Pengutronix. Although ICECC should be a good tool to distribute the computing power to a cluster of machines, it recently turned out that things are not that simple.