Showcase: Graphics on i.MX8MP

Enabling the graphics output pipeline on the i.MX8M Plus (i.MX8MP for short), is the most recent example on how open-source and upstream driver support for GPU and display engines can reduce effort and risk in a new project.

Graphics driver background

Graphics support in embedded devices has changed quite a bit over the last few years. Let's dive a bit into the history of graphics hardware and the accompanying software support to set the scene.

Historically, embedded devices would only have very minimal graphics support, if at all. They could drive simple LC displays and all the work of actually producing the picture to show on the display had to be handled by the CPU, with no hardware acceleration whatsoever. On one hand, this was convenient, as the CPU is very flexible in what it can do and driver support for the simple hardware units driving the displays didn't require much effort: all they needed to do was to get a piece of bitmap memory filled by the CPU out on the display data bus with the right timing. On the other hand, the limited processing power available on the CPU set very strict bounds on what was possible to do on the graphics side without impacting the primary use of the device, as most devices are built to fulfill other jobs besides displaying pretty graphics.

Around 15 years ago Moore's law still progressed at full speed and it was finally possible to fit real Graphics Processing Units (GPUs) into the space, power and cost envelope required in the embedded space. With GPUs taking over most of the actual work of drawing the content to show on the display, the CPU was free again for other jobs and while at the same time providing much richer user interfaces.

This however came with a new cost: the complexity of the software needed to drive those GPUs is a huge step up from the simple drawing implementation on the CPU. Also, the GPU marketing segment is very competitive, with multiple companies trying to sell GPUs for use in embedded systems. In this environment, most of those companies favored a closed-source approach for the driver software, in order to not reveal any secrets that may set them apart from the competition.

Issues with closed-source drivers in embedded devices

Embedded systems have some unique requirements in that they are usually designed to do a very specific task at the lowest possible price point. So there is little room to compensate inefficiencies in the processing pipelines with raw compute power. Thus the software has to be optimized to provide the most efficient processing pipelines possible. Also, most of them are designed to be used for a very long time. With more and more of those devices connected to the network, this means it must be easy to update the software running on them in order to be able to fix functional or security issues.

Both of those requirements are very hard to meet when depending on closed source components that one has no direct control over. Even if the GPU vendor is able and willing to provide support, changes to the driver software can take a very long time, leading to failures to meet the window of opportunity where a device can be brought into the market. Having to take into account the requirements of the closed-source bits in the system may also prevent timely updates. This may apply to new system components like kernel and base libraries in the system when those components leave their support period.

Closed-source components might also bring in assumptions and dependencies that severely limit the ability to alter the software stack later on in the lifetime of the device.

Alternative solutions

The Pengutronix graphics team has an extensive track record in building open-source drivers for these hardware units.

On the GPU side, this journey started when some of our customers were not willing to accept the risk of having a third-party closed-source component at the core of their product. They worked with Pengutronix to bring to life a open-source alternative. The NXP i.MX line of SoCs is very popular in the industrial and medical embedded ecosystems, as it ticks a lot of the required feature boxes. Starting with the i.MX6 those SoCs include GPUs from VeriSilicon (formerly Vivante). Some volunteers in the free software community had already started a reverse-engineering effort targeting these GPUs, called Etnaviv. Pengutronix jumped on-board, helped to get it into shape and provide a commercial quality open alternative to the closed-source vendor driver.

Today the Etnaviv driver stack is in use in a breadth of embedded devices, from medical appliances to in-flight entertainment systems to automotive and industrial control systems. While Pengutronix, as a strong partner with a really deep technical expertise in this area, is always standing by to help you along if the need arises, all the components are available as open-source and can be used, extended and improved by anyone, eliminating the risk of vendor lock-in for this critical component.

The Etnaviv driver stack integrates in the upstream frameworks like DRM in the Linux kernel and Mesa on the userspace side, so it's quite close in features and behavior to the general Linux desktop graphics stack, making embedded projects less special.

i.MX8MP GPU enablement with upstream drivers

Recently, the Pengutronix graphics team was tasked with enabling the GPU on the NXP i.MX8M Plus, which is currently in pre-production. This SoC is very interesting to many industrial customers, as the i.MX8M line is a successor to the really popular i.MX6 series. A lot of the software support, written and refined over the last 10 years, can be reused, significantly reducing the effort to get the basic system functionality working. The i.MX8MP also packs some interesting features like CAN FD and TSN Ethernet.

The GPU is a slightly different variant of the GC7000L, found in the already supported i.MX8MQ SoC. While some small adaptions were needed to run the open-source driver stack on this specific GPU, it took the Pengutronix graphics team less than a week total effort to get the GPU up and running. Years of experience working on those parts of the software stack really helped in getting this working quickly.

If you are planning to start a new project involving graphics input/output support we are happy to talk to you!

Back to technical showcases

Further Readings

Showcase: Preempt RT and Time Sensitive Networking

Nowadays, even small and cheap microcontrollers offer enough calculation power to perform time critical tasks within an industrial environment. However, as soon as actors and sensors are spread over an entire facility and are to be connected over Ethernet, the actual moment when a data packet will get processed becomes very hard to predict. At this point, Linux running a Preempt RT Kernel altogether with a network featuring Time Sensitive Networking (TSN) capabilities can help.

Showcase: Embedded off-the-shelf

A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.

Showcase: Continuous Testing

About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.

Showcase: Remote Working

Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.

Yes we CAN... add new features

Have you ever experienced an otherwise fine product that is missing just the one feature you need for your application?

DSA in Barebox

The v2022.05.0 Release of barebox introduced initial support for the Distributed Switch Architecture (DSA) Framework. DSA is originally a subsystem from the Linux Kernel, which exposes the individual ports of a network switch IC as virtual network interfaces.

First Steps using the candleLight

So you went and got yourself one of our fancy rocket-penguin branded CandleLight dongles or, being the die hard hacker you are, went and soldered one up in your toaster oven labeled "not food safe". What's next then? How do you use this thing? Let's answer these question by grabbing a Raspberry Pi and exploring some of the possibilities.

Pengutronix at Embedded World 2020

Yesterday, Embedded World started, in normal times one of the largest trade shows for embedded development in Europe. While many exhibitors (and thus maybe also lots of visitors) have canceled their presence due to the coronavirus, we present our booth and our demo show cases as usual.

Pengutronix at FrOSCon 2018

This year, a team from Pengutronix attended FrOSCon in St. Augustin for the first time. We took the opportunity to shake hands, talk about our latest developments and meet hackers interested in working with embedded Linux.

Pengutronix TechWeek 2018: NIR for Etnaviv

Once in a year, mostly during springtime, no phone call is answered at Pengutronix, no customer Mail receives a reply, no BSP is built and no coffee machine gargles. But, no fear, this is not the end! It is just the time for something new! New ideas, focused development on new world domination plans and time to think outside the box. Yes, it's again the time for the Pengutronix TechWeek!