#system on modules

Amazon acquires FreeRTOS       FreeRTOS is a real-time operating system kernel for embedded devices. In development since 2003, it is the market-leading real-time operating system, having been ported to more than 40 microcontroller architectures. On November 29, 2017, Amazon acquired FreeRTOS and announced Amazon FreeRTOS (or a:FreeRTOS), a real-time operating system kernel based on FreeRTOS and focused on low-power, connected devices.

Amazon seems to be interested in integrating the most widely used real-time operating system — which, consequently, has a very broad user base — with their cloud services, like AWS IoT Core and AWS Greengrass. After the acquisition, Amazon introduced version 10 of FreeRTOS. Along with new features like cloud and edge connectivity libraries, the kernel license was changed from a modified version of the GNU General Public License (GPL), which the FreeRTOS project used previously, to the more permissive MIT license. This means that FreeRTOS is still a separate product and can be used freely, with or without Amazon services.

Why this is relevant: Heterogeneous Multicore Processing (HMP)

Toradex offers the Colibri iMX7 and Colibri VF61 System on Modules (SoMs) / Computer on Modules (CoMs). These products are based on the NXP® i.MX7 and Vybrid SoC respectively. The processors on these modules feature an Arm® Cortex-A core, usually running an operating system such as embedded Linux; and an Arm Cortex-M4 microcontroller core, which can run bare-metal code or a real-time operating system like FreeRTOS. Here's an outline of use cases:

 Control      systems:      The Cortex-M4 core can be used as a controller for a dynamic system,      running, e.g., a PID controller, offloading the Cortex-A core running      Linux, which, in turn, could be displaying a human-machine interface.

 Data      acquisition:      On applications that make use of an analog-to-digital converter, the      Cortex-M4 core can be employed to acquire data at a high sample rate more      easily, also offloading the other core.

An example application using the Toradex Colibri iMX7 System on Module is a self-balancing robot named TAQ. It was developed in partnership with Antmicro and Qt. The robot runs embedded Linux on the Cortex-A7 core, presenting the robot's face animations and sensor data received from the Cortex-M4 core. The Cortex-M4 core runs FreeRTOS, acquiring data from sensors such as an accelerometer and a gyroscope. It uses this data to control the dynamic system, employing a PID controller and a Kalman filter. The cores communicate via RPMsg. A kernel module on the Linux side creates a virtual serial interface which connects to the Cortex-M4; data exchange can be achieved by writing to or reading from this serial device, just as it would be done with a "real" serial device.

The sources for this demo are open and can be found here (Linux side, user interface) and here (Cortex-M4 firmware).

The Toradex Colibri iMX7 was also used on a tablet demonstrating some low-power applications. This video showcases the demo, which features sensor data acquisition and visualization while independently displaying power consumption on a separate LCD display.

Toradex supports and maintains a FreeRTOS repository for its System on Modules. Many peripheral drivers and examples are included and ready to use. Some documentation is available at the Toradex Developer Center showing how you can get up and running with the built-in FreeRTOS examples. Watch our webinars for deeper insight into heterogeneous multicore systems development.

What about the future?

On the software side

There are many new players in the RTOS market. In 2016, Wind River Systems and the Linux Foundation launched Zephyr, a real-time operating system also focused on small, connected devices, targeting the IoT market. The Toradex Colibri iMX7 System on Module is officially supported by Zephyr, and development is ongoing.

On the hardware side: the all-new NXP i.MX 8-based Apalis and Colibri SoMs

Toradex has just launched the early access phase of its brand-new System on Module, the Apalis iMX8. This powerful SoM is based on the NXP i.MX 8QuadMax (i.MX 8QM) applications processor, featuring two Arm Cortex-A72 cores, four Cortex-A53 cores and two Cortex-M4 cores. Packed with 4 GB of LPDDR4 RAM and 16GB of eMMC flash storage, it also has built-in dual band Wi-Fi (802.11ac) and Bluetooth (Bluetooth 5-ready). This System on Module is ideal for computer vision applications: there are two integrated Vivante GC7000 GPUs, supporting Vulkan and OpenGL ES.

The two Cortex-M4 cores open up many possibilities: as an example, you can run two different applications or real-time operating systems independently, to balance a data processing application; or use one core for sensor data acquisition and the other for post-processing. The microcontroller cores can also be used as a safety fallback for a graphical display, or to present early boot animations instead of static images.

Toradex presented an Apalis iMX8 demo at Embedded World 2018. Watch this video to check it out, along with some other exciting partner demos.

Soon: Colibri iMX8

Toradex and NXP® invite you to this introduction of the new Colibri SoM based on the NXP i.MX 6ULL System-on-Chip. The Colibri System on Module effectively showcases NXP’s power-efficient and cost optimized applications processor – the i.MX 6ULL. Toradex’s Colibri iMX6ULL features Wi-Fi and Bluetooth, secure encryption, and a wide range of other interfaces for connecting peripherals such as sensors, GPS, displays and camera sensors.

In this webinar, we break down all the essential details of this exciting new SoC and SoM in Toradex’s proven Colibri form factor. We will show you benchmarks with other NXP i.MX 6 and i.MX 7 based Colibri modules, in order to familiarize you with the positioning for different applications.

Visit our webinar page to grab your free registrations now: https://www.toradex.com/webinars/colibri-nxp-imx6ull-system-on-module-introduction

Date and Time:

Slot 1 - Wednesday, November 29, 2017 02:00 PM - 03:00 PM (CET)

Slot 2 - Wednesday, November 29, 2017 10:00 AM - 11:00 AM (PDT)

Speakers:

Mario Centeno, Global Product Manager, Low Power i.MX, NXP

Daniel Lang, CMO, Toradex

Toradex is excited to announce the launch of the new OV5640 5MP MIPI-CSI camera module. This is an add-on board for the Apalis family of system on modules (SoMs) compatible with the Ixora carrier board.

The CSI camera module uses an OmniVision OV5640 camera sensor with built-in autofocus. The OV5640 is a low-voltage, high-performance, 1/4-inch, 5-megapixel (2592x1944) CMOS image sensor.

Key features include:

Support for video capture at up to 120 frames per second (FPS) at 320x240, 30 FPS at 1080p or 15 FPS in the full 5-megapixel resolution

Direct connection to the MIPI Camera Serial Interface 2 (MIPI CSI-2) on Apalis carrier boards using a 24-way 0.5 mm pitch FFC cable

No requirement for an external power supply

Compatibility with Toradex Yocto Project-based Standard Linux  

Compatibility with Windows Embedded Compact (WinCE) 7 and 2013

Compatibility with the complete Apalis family including NVIDIA T30, NVIDIA TK1, NXP i.MX 6 and the upcoming NXP i.MX 8-based modules

                    OV5640 5MP CSI camera module, front and back views

“The new camera module makes it easier for Toradex customers to prototype computer vision projects. Our Apalis computer modules are a proven platform for embedded vision applications. Technologies such as deep learning and frameworks like OpenCV, Tenserflow and Caffe are progressing quickly and enabling more and more possibilities in edge devices,” says Daniel Lang, Chief Marketing Officer for Toradex.

OV5640 5MP CSI camera module connected to the Ixora carrier board

The module is globally available in the Toradex Webshop. As with all Toradex products, you’ll also benefit from excellent support via channels like the Developer Centre and the Toradex Community, as well as direct support from Toradex offices around the globe.

For more information about the MIPI-CSI camera module, visit our Developer Centre.

To learn more about the Toradex Apalis ecosystem, visit the Apalis overview page.

Meet our engineers this month at the following events:

Qt World Summit 2017, Germany, Oct 10-12, 2017

IoT Japan 2017, Japan, Oct 11-13, 2017

Arm TechCon Expo 2017, United States, Oct 24-26, 2017

About Toradex:

Multi-Camera 3D Surround View systems are already established in many passenger vehicles. Demand for such systems now also grows for special vehicles like construction machines, harvesters, buses or military vehicles. In comparison to personal cars, these vehicles are produced in a rather moderate volume and have a high need for customization of the optical system.

In this blog post, we present a customizable 3D Surround View Turn-Key solution based on the Toradex Apalis iMX6 SoM based on the NXP® i.MX 6 SoC, that can be quickly adapted concerning the specific requirements of the targeted vehicle. 

Please watch this video from Embedded World 2017 in Nuremberg, Germany, where this system solution was presented:

We also explain how to get familiar with the topic of 3D surround view, evaluating the solution via a step-by-step approach and finally how to come to a product meeting your specific needs.

TES 3D Surround View Stitching Library – the heart of the system solution

TES Electronic Solutions provides a 3D Surround View software library designed for embedded systems, CPU/GPU independency, and high customizability. Detailed information can be found on the TES 3D Surround View Website.

The library takes raw fish-eye pictures from memory and computes a 3D View of the vehicle in its maneuver situation.

TES 3D Surround View Highlights:

Increases safety and comfort through improved situation awareness:

Excellent stitching and 3D scene generation

Watch your maneuver situation from any position outside or inside the vehicle

Any number of predefined viewing positions (“Views”) possible with animated transitions between different views

Many parameters configurable per “View”

Highly customizable and portable

Independent from the used type of cameras (the software expects raw fish-eye pictures in memory)

3D Surround View Turn-Key Solution

In the following, we present a complete HW/SW system solution based on the Toradex Apalis iMX6Q IT Module allowing easy hardware customization to meet specific customer demands.

System Setup

The System setup consists of the following components: Processor:

Toradex Apalis iMX6 IT Computer On Module with Heatsink

Toradex Ixora Carrier Board

Camera System:

First Sensor Mobility Area View Kit – DC–AVK including

4x 1MPixel MJPEG Ethernet Cameras (PoE), DC3K-1-EAP

4x 5m Ethernet cables (up to 25m possible for each camera)

Micrel Gigabit Ethernet (PoE) Switch

Metal Plate for Desktop Application, additionally 4 Vacuum cups for vehicle application included

Software:

TES 3D Surround View Software under Linux on Apalis iMX6

Resource Usage:

The 3D Surround View algorithm mainly exploits the GPU. The CPU is only used for system control and in the case of view-changes.

Configuration: 4 MJPEG IP cameras, combined 80Mbit data rate @30 fps and 1Mpx resolution. Low polygon car model, 1280x1024 screen resolution at 60Hz refresh rate.

GPU: 3-6 GFlops (proportional to pixel amount depicted on screen)

RAM usage: 60 MB in total

4* 10 MB for camera data buffers

2 MB for 3D environment and car model

18 MB for camera video textures

This turn-key system solution can be rapidly reproduced for evaluation and prototyping purposes. Of course, also the separate components can be exchanged or adapted to the target application needs, e.g.:

The cameras can be exchanged against other types.

A dedicated application specific main board can be designed taking the Ixora Carrier board as a reference.

The user interface and HMI overlay of the 3D Surround View software can be easily adjusted.

The 3D model of the vehicles can be easily exchanged.

Both Toradex and TES, are open to provide the necessary support. Getting started and familiar with 3D Surround View 3D Surround View is not only a topic of running an algorithm on an embedded platform and be done with it. There are many aspects of the system setup that need to be considered and require customization for specific use cases, e.g.:

Selection of the appropriate cameras regarding lens, resolution, physical data transmission (cable length!), power supply, number of cameras etc.

Positioning of the cameras (usually the higher the better)

Defining the appropriate viewing positions, i.e. virtual position and viewing vector of the “observer” in the 3D scene

Optimizing the 3D Surround View parameters, e.g. Environmental model (shape and size of the projection bowl), stitching method and parameters.

User interface for the 3D Surround View System

3D Model of the vehicle, potentially with transparency.

Just consider an excavator compared to a personal car: On an excavator, you need significantly longer cables to the cameras, the cameras are mounted typically much higher. Latency is not that much of a concern as the driving speed is lower and maybe you just want to apply only 3 cameras or mount the cameras un-symmetrically because the driver can see the left front part of his vehicle very well while he cannot see the right front part at all.

So it might be a good idea to first “play” with a 3D Surround View system to get an idea of such application aspects. For this purpose or for those who just want to give the TES software a try there is an offline PC demonstrator available at zero cost. TES provides this demonstrator on request in a zip archive with no need of installation (just unpack). Any modern Windows PC should be sufficient to run this software. The program comes with prerecorded MJPEG Ethernet streams in *.PCAP-format and runs these streams through the TES 3D Surround View software. The PC keyboard and mouse can be used to change the viewer position: predefined views are available on the number keys and “free-flight” is possible via mouse and (Shift-) arrow keys. Also, some of the 3D Surround View system parameters like the bowl-size and -shape as well as the stitching parameters can be adjusted. Do not hesitate to get in touch with TES (coordinates below) to get this Software.

First steps in the Lab and on the vehicle

Now that you have played with the software (and assuming it does more or less what you need) you might want to give it a try with real-life cameras, either in lab conditions or right away on the vehicle.

The fastest way to do that is to get a “First Sensor Mobility Area View Kit – DC–AVK” (contents see above under “System Setup”) and the TES 3D Surround View Windows PC Software in Online-Mode (Evaluation Version). This kit allows lab setups as well as on-vehicle installation using a Windows Laptop. Up to 20m cable length (between cameras and switch) has been tested so far. Also here TES can assist you in purchasing this setup.

And now on the embedded hardware…

Having tested the system under lab conditions and on the vehicle that target platform aspects now get focus (again). I.e. you want to see your setup running on the potential target hardware. At that point, you may want to take your Apalis iMX6 module with Ixora carrier board (or buy some from Toradex if you do not have those, yet) and get the TES 3D Surround View software for that target platform (prototype version). I.e. you get rid of the PC and use the Apalis/Ixora platform instead. The TES prototype software for that target platform has the same functionality as the evaluation version for PC and you can re-use the configurations that you have setup at the earlier stage now on the target.

Getting to a product

For a product, you now have to customize the system to meet your specific application requirements. I.e. you use the Apalis iMX6 module as core and might re-design the carrier board based on the Ixora design data to meet your specific requirements regarding, interfaces, power supply, mechanics etc.

In that step or even before you also may want to exchange the cameras with other types.

Again, TES can assist you also in the hardware and system customization, development and (if wanted) manufacturing process.

Summary