Showcase: Remote Working
Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.
In this article we would like to present to you our Remotelab infrastructure. It was initially developed for Continuous Testing, but has now shown to be one of our main enablers for remote working, allowing us to stay productive with less than a handful of colleagues on-premise.
Our Remotelabs consist of racks housing embedded prototype boards by our customers. Additional infrastructure in our Remotelabs enables our developers to remote control these boards (see below). Once the hardware is located in our infrastructure, it can also be tested, automatically and continuously. This allows us to be alerted about any breaking changes affecting our customers.
A few words about continuous testing
In fact, the ability to test permanently and automatically is one of the main drivers for our ongoing development of labgrid (the software framework to access the hardware) and the professionalization of our Remotelab infrastructure.
We would like to show you the advantages of continuous testing in our Showcase: Continuous Testing.
How can we support you?
As a service provider in the software development environment, we are not in the position of selling you complete and ready-made Remotelab infrastructure, but we would be happy to assist you in such projects: Whether it is about integrating support for your test hardware in labgrid, working out test strategies, developing tests, or integrating labgrid into your processes – do not hesitate contacting us.
Even without investing in your own Remotelab, you can still benefit from our Remotelab infrastructure, as our customers hardware is only a few keystrokes away for our development team at any point in the product cycle.
In the last years we have constructed several Remotelabs and want to share our design process with you.
What components does a Remotelab consist of?
19"-Rack
The supporting component of our Remotelab: We use a 19" TE network cabinet from Rittal. The transport rollers are a particularly useful feature. To protect the hardware, each shelf is covered with an ESD shelf lining. In addition, cable guide rails enable an (at least initially) tidy Remotelab. Two suitable sockets for rack mounting provide the necessary power for the embedded boards.
Network switch
Our favorite in this category is the CBS350-24P-4G from Cicso, as it is PoE capable, silent and managed. By using PoE we want to minimize the need for individual power supplies in future in-house hardware developments. The use of managed switches allows us to respond to any special requirements the customer hardware may have for the network setup.
Quiet operation of the switches is important to us, as some of the Remotelabs are located in the developer offices.
Small Server
A small server is needed to manage each of the devices in the Remotelab. The hardware requirements for this labgrid node are quite low, aspects we pay attention to are:
- Quiet operation -- Our Remotelab servers use PC Engines APU.2E4, allowing fanless operation.
- Stable USB -- Large USB installations, such as our Remotelabs are, do, in our experience, carry a lot of potential for frustration. The APU.2E4's internal USB controllers and the EXSYS EX-49010-2 controllers installed in addition have proven to be comparatively reliable in operation.
- CAN bus integration -- For communication with our own custom hardware and for testing the embedded devices, we use CAN-FD capable miniPCIe cards from Peak System, providing four independent CAN buses.
Switchable socket
Have you tried turning it off and on again? -- The easiest way to restart the hardware is to toggle the supply voltage, for example via network-connected power strips, which can be controlled using labgrid. As a low-cost alternative for smaller setups, we are also gaining initial experience with individual sockets controllable via WiFi.
For use in the Remotelab, where many devices are to be controlled, we use GUDE 8080 socket strips, for individual devices we are experimenting with individual sockets from the manufacturer SONOFF, which are flashed with the free Tasmota firmware.
USB-Hubs
USB is almost inevitable in embedded development, we use it to communicate with the hardware and to upload the software. When looking for reliable USB hubs, it is important to try products from different manufacturers. We have had good experiences with the 10-port UA0096 hubs from LogiLink.
An interesting use case that is unfortunately not covered by these hubs is switching the supply voltage of individual ports on and off using uhubctl.
WiFi router
In order to be able to test Bluetooth and WiFi functionalities of the embedded hardware, each remote lab has its own WiFi router, which has been flashed with the free OpenWRT firmware. This router is not connected to the Internet or any other network and is only used to test the connection setup in automatic tests.
Serial console
For the connection of the serial consoles of the embedded devices we use simple USB-serial converters, on which preferably a CP210x chip from Silicon Labs is used. This chip allows us to store a unique ID in each USB-serial converter easily. We put a note containing this ID on any USB-serial converter, helping us managing a few dozen converters per lab.
Remotely setable jumper pins
In order to be able to set and release reset or boot option jumpers as flexibly as possible, potential-free switches are helpful; here we rely on our own developments connected via OneWire or CAN. Alternatively, USB-controlled relays or professional Modbus devices are also available and usable in labgrid.
Access from remote: Our labgrid Integration
To be able to take full advantage of the Remotelab, the hardware alone is not enough. For the control of the embedded boards from the developer PC and as a basis for tests and development on real hardware, Pengutronix uses the open source tool labgrid as an abstraction layer.
Labgrid provides for automated tests as well as for interactive work an intuitive and at the same time versatile interface for working with Embedded Linux devices.
There are several ways to access embedded boards via labgrid, e.g.:
- Serial console -- Almost always available, even early in the boot process.
- Access via network and ssh -- More convenient especially later in the development process.
- Camera stream of the display -- Relevant during driver development and development of graphical applications.
How does the software get onto the embedded board?
This depends on the corresponding hardware and what is to be tested. For example, the bootloader can load the data via the network, a programming adapter can be connected or the software gets loaded via USB. If possible, however, we use the USB-SD-Mux, that allows switching a micro SD card between the Remotelab server and the prototype hardware.
Back to technical showcasesFurther Readings
Showcase: Embedded off-the-shelf
A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.
Showcase: Continuous Testing
About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.
labgrid is going on a live tour!
labgrid makes it possible to remote-control embedded linux devices and to implement integration tests of a complete embedded Linux system on real hardware. The Pengutronix developers and other companies have been using labgrid as a centerpiece of their embedded software development infrastructure to great success for quite some time now.
Linux Automation Test Automation Controller: A one Device labgrid Exporter
Our subsidiary Linux Automation GmbH introduces the LXA TAC (Linux Automation Test Automation Controller): an all-in-one labgrid exporter. The LXA TAC offers the usual interfaces to control one or more embedded devices (DUTs, devices under test) interactively or automatically with labgrid.
Yes we CAN... add new features
Have you ever experienced an otherwise fine product that is missing just the one feature you need for your application?
DSA in Barebox
The v2022.05.0 Release of barebox introduced initial support for the Distributed Switch Architecture (DSA) Framework. DSA is originally a subsystem from the Linux Kernel, which exposes the individual ports of a network switch IC as virtual network interfaces.
Pengutronix at Embedded World 2022
Welcome to our booth at the Embedded World 2022 in Nürnberg!
First Steps using the candleLight
So you went and got yourself one of our fancy rocket-penguin branded CandleLight dongles or, being the die hard hacker you are, went and soldered one up in your toaster oven labeled "not food safe". What's next then? How do you use this thing? Let's answer these question by grabbing a Raspberry Pi and exploring some of the possibilities.
labgrid Tutorials
This week, we started our series of YouTube labgrid tutorials. In the next few weeks we will publish more video tutorials showing you labgrid's features and giving you handy tips and tricks.
The LXA IOBus line of lab automation devices
I would like to present to you the LXA IOBus, a CAN-based ecosystem consisting of a protocol, a gateway server and new class of Linux Automation GmbH devices, including the Ethernet-Mux and the 4DO-3DI-3AI input/output board.