Showcase: Continuous Testing

About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.

How can we support you?

By using Continuous Testing (CT) on real hardware, we are able to detect and fix occurring problems at an early stage. Our development projects essentially consist of two phases:

Initial Product Development

Many projects start with an initial development phase: In this phase we support you in commissioning your hardware, as well as building a Board Support Package (BSP), which is exactly tailored to your hardware. In this phase we develop for example missing drivers, add missing functions to software components or optimize the boot time of your device. In this phase software components are usually constantly updated to the latest versions. At the same time we offer to develop tests for your hardware and your BSP. These tests check the core functions of the system during development. This phase usually ends with the release of your product.

Long Term Maintenance

In addition, we offer you a permanent maintenance of your BSP: We regularly (e.g. several times a year) update your system and ensure with the tests that the functions are still given.

The subsequent maintenance provides you with significant advantages: If a software component should cause an error in your product or a security gap should occur, you will always have an up-to-date system at your disposal. This enables you to have short release cycles. At the same time, the changes from update to update are smaller, so that the actual effort for maintenance is lower.

Please do not hesitate to contact us: In combination with your product knowledge, we will come up with a development and maintenance concept for your operating system components that is tailored to your requirements.

Real World Example

In the following example, a customer relies on Verified Boot for its product. A chain of cryptographic operations ensures that only an operating system signed by the manufacturer can be run on a device. Each component (e.g. the boot loader) checks whether the next software component to be loaded (e.g. the Linux kernel) has a valid signature. Only if this is the case, the respective next component is executed.

Further Information on Verified Boot

Our colleague Marc Kleine-Budde gave the overview talk "Verified Boot: From ROM to Userspace" at ELC Europe 2016. (Youtube, Slides)

The Verified Boot chain, from the configuration of the ROM code of the processor, over the boot loader, the Linux kernel up to the user space, were integrated into the BSP and are still being maintained by Pengutronix.

Test Case

To ensure that the verification of both the FIT image (including Linux kernel and device tree) and the dm verity protected file system work, they are tested. To do this, the FIT image is modified and an attempt is made to boot the system. In a different test, the file being executed as the init process in the dm-verity-protected file system is modified. Reading and executing this file is expected to fail, since the dm-verity hash tree is corrupt. In both scenarios, the boot attempt fails, thus Verified Boot works as expected.

During this test, various functions of the barebox boot loader are tested as a side effect: Images are copied between partitions, file systems are mounted and modified and the triggering of the watchdog is tested.

Error Case

During a maintenance update of the BSP, the boot loader has now been updated and the previously discussed test cases failed. The good news: Verified Boot still worked: The modified system was not executed.

But: The test revealed two bugs in barebox: One bug occurred when copying to POSIX device-files (Fix by Sascha Hauer). The other bug concerns the watchdog (Fix by Ahmad Fatoum). It is expected to reset the system after the Verified Boot chain was corrupted.

Both errors were uncovered by the test suite and the appropriate fixes were integrated into the BSP before the new BSP version was delivered to the customer.

Back to technical showcases

Further Readings

Showcase: Embedded off-the-shelf

A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.


Showcase: Remote Working

Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.


labgrid Tutorials

This week, we started our series of YouTube labgrid tutorials. In the next few weeks we will publish more video tutorials showing you labgrid's features and giving you handy tips and tricks.


Update of our Remotelab equipment

If it looks like an advertising blogpost, reads like an advertising blogpost ... it probably is an advertising blogpost! Nobody likes to read advertisements and we don't like to write them at all, but like all proud parents, we would like to show you the new products that our corporate subsidiary, Linux Automation GmbH, has freshly added to their store. With these new products we, and maybe soon you, will complete (y)our Remotlab infrastructur.


The LXA IOBus line of lab automation devices

I would like to present to you the LXA IOBus, a CAN-based ecosystem consisting of a protocol, a gateway server and new class of Linux Automation GmbH devices, including the Ethernet-Mux and the 4DO-3DI-3AI input/output board.


Showcase: Fail-Safe (OTA) Field Updating

Enrico Jörns | | didyouknow, rauc

Being able to robustly and securely update embedded systems and IoT devices in the field is a key requirement of every product today. The update framework RAUC is the basis for a modern and future-proof solution. In this showcase we present the basic principles of a fail-safe update system and how Pengutronix can support you with implement this for your platform.


Showcase: Graphics on i.MX8MP

Enabling the graphics output pipeline on the i.MX8M Plus (i.MX8MP for short), is the most recent example on how open-source and upstream driver support for GPU and display engines can reduce effort and risk in a new project.


Pengutronix at FOSDEM 2021

"FOSDEM is a free event for software developers to meet, share ideas and collaborate. Every year, thousands of developers of free and open source software from all over the world gather at the event in Brussels. In 2021, they will gather online." -- FOSDEM


Pengutronix at Live Embedded Event

conference, event, testing

Now that, due to the COVID-19 pandemic, everyone has gotten used to digitalisation and online conferences - it has never been easier to organise a conference and bring together all experts and interested parties for a few hours of intensive exchange of ideas on a certain topic.


USB-SD-Mux: Automated SD-Card Juggler

Once the bootloader on your embedded device is up and running the development of kernel and userland in PTXdist-based BSPs is usually based on booting from network. Thus there is no need for the developer to write the boot media with a new image.