Showcase: Continuous Testing
About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.
How can we support you?
By using Continuous Testing (CT) on real hardware, we are able to detect and fix occurring problems at an early stage. Our development projects essentially consist of two phases:
Initial Product Development
Many projects start with an initial development phase: In this phase we support you in commissioning your hardware, as well as building a Board Support Package (BSP), which is exactly tailored to your hardware. In this phase we develop for example missing drivers, add missing functions to software components or optimize the boot time of your device. In this phase software components are usually constantly updated to the latest versions. At the same time we offer to develop tests for your hardware and your BSP. These tests check the core functions of the system during development. This phase usually ends with the release of your product.
Long Term Maintenance
In addition, we offer you a permanent maintenance of your BSP: We regularly (e.g. several times a year) update your system and ensure with the tests that the functions are still given.
The subsequent maintenance provides you with significant advantages: If a software component should cause an error in your product or a security gap should occur, you will always have an up-to-date system at your disposal. This enables you to have short release cycles. At the same time, the changes from update to update are smaller, so that the actual effort for maintenance is lower.
Please do not hesitate to contact us: In combination with your product knowledge, we will come up with a development and maintenance concept for your operating system components that is tailored to your requirements.
Real World Example
In the following example, a customer relies on Verified Boot for its product. A chain of cryptographic operations ensures that only an operating system signed by the manufacturer can be run on a device. Each component (e.g. the boot loader) checks whether the next software component to be loaded (e.g. the Linux kernel) has a valid signature. Only if this is the case, the respective next component is executed.
The Verified Boot chain, from the configuration of the ROM code of the processor, over the boot loader, the Linux kernel up to the user space, were integrated into the BSP and are still being maintained by Pengutronix.
To ensure that the verification of both the FIT image (including Linux kernel and device tree) and the dm verity protected file system work, they are tested. To do this, the FIT image is modified and an attempt is made to boot the system. In a different test, the file being executed as the init process in the dm-verity-protected file system is modified. Reading and executing this file is expected to fail, since the dm-verity hash tree is corrupt. In both scenarios, the boot attempt fails, thus Verified Boot works as expected.
During this test, various functions of the barebox boot loader are tested as a side effect: Images are copied between partitions, file systems are mounted and modified and the triggering of the watchdog is tested.
During a maintenance update of the BSP, the boot loader has now been updated and the previously discussed test cases failed. The good news: Verified Boot still worked: The modified system was not executed.
But: The test revealed two bugs in barebox: One bug occurred when copying to POSIX device-files (Fix by Sascha Hauer). The other bug concerns the watchdog (Fix by Ahmad Fatoum). It is expected to reset the system after the Verified Boot chain was corrupted.
Both errors were uncovered by the test suite and the appropriate fixes were integrated into the BSP before the new BSP version was delivered to the customer.Back to technical showcases
A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.
Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.
If it looks like an advertising blogpost, reads like an advertising blogpost ... it probably is an advertising blogpost! Nobody likes to read advertisements and we don't like to write them at all, but like all proud parents, we would like to show you the new products that our corporate subsidiary, Linux Automation GmbH, has freshly added to their store. With these new products we, and maybe soon you, will complete (y)our Remotlab infrastructur.
I would like to present to you the LXA IOBus, a CAN-based ecosystem consisting of a protocol, a gateway server and new class of Linux Automation GmbH devices, including the Ethernet-Mux and the 4DO-3DI-3AI input/output board.
Being able to robustly and securely update embedded systems and IoT devices in the field is a key requirement of every product today. The update framework RAUC is the basis for a modern and future-proof solution. In this showcase we present the basic principles of a fail-safe update system and how Pengutronix can support you with implement this for your platform.
"FOSDEM is a free event for software developers to meet, share ideas and collaborate. Every year, thousands of developers of free and open source software from all over the world gather at the event in Brussels. In 2021, they will gather online." -- FOSDEM
Now that, due to the COVID-19 pandemic, everyone has gotten used to digitalisation and online conferences - it has never been easier to organise a conference and bring together all experts and interested parties for a few hours of intensive exchange of ideas on a certain topic.
Once the bootloader on your embedded device is up and running the development of kernel and userland in PTXdist-based BSPs is usually based on booting from network. Thus there is no need for the developer to write the boot media with a new image.