This hotly debated topic has been around for decades, and it is just as alive today as it was 28 years ago. The truth is, there are fundamental differences in the theory which drives the design of a monolithic kernel versus a microkernel. In this post, I will extrapolate from my knowledge of various kernel designs to explore what these two primary types are, what their features, benefits, drawbacks and implications may be. I’ll also briefly explore the extension of the theories that drive these designs into “fringe” types of kernels (such as a nanokernel).

Key Concepts

Before hopping into the discussion about the differences between these two kernel archetypes, I need to lay out some key concepts which will help bound the discussion.

Userspace vs. Kernelspace

Broadly, userspace is considered to be “outside” of the kernel. Kernelspace is “inside.” The way this is typically implemented is through the use of multiple address spaces, aided by the hardware (MMU, MPU). This usually results in all userspace applications having believing they have the whole address space to themselves, while in reality the MMU is remapping some Virtual Addresses (VAs) to the real Physical Addresses (PAs). Actions in userspace cannot (well, SHOULD not) perturb the operation of the core kernel. Contrast this to kernelspace, wherein any part of the kernel is available to poke and tinker with from code running in this space.

This separation of spaces may also use other hardware features (Rings in x86-land, Exception Levels in ARM) to further represent the distinction and privilege of code running in each space.

Examples of userspace are your typical Linux application.

Examples of kernelspace are a Linux .ko kernel module.

Kernel-User Boundary

This boundary is the line which separates the userspace and kernelspace. Crossing this boundary involves a change in privilege level, and is often accomplished by invoking a system call. Why and when we cross this boundary is a key point of interest in the discussion of kernel design since it is usually an expensive operation.

Essential Operating System Functions

For the purpose of this discussion, I consider the following to be essential to any OS for it to be of any use.

• Memory Management
  • Addresses needs of the kernel to map/unmap/remap Virtual Memories to Physical ones
• Task/Process Management
  • Addresses needs of the kernel to handle multi-tasking
• Inter-process Communication (IPC)
  • Addresses needs of kernel and applications to communicate via a common interface
• Device Drivers
  • Addresses needs of applications to communicate with the outside world
• Filesystems
  • Addresses needs of application and kernel to store/retrieve data via a common interface

Monolithic Kernels

Monolithic kernels and the essential properties which make it monolithic can be succinctly summed up in the following phrase:

Everything except the application exists in kernelspace.

In a monolithic kernel, all the essential components, and many other accessory components, live in kernelspace.

The design choice to bring all these functions and services into the kernelspace has several benefits, drawbacks, and implications.

Benefits

Since all the code that directly interacts with devices live in the same address space, moving data around is an inexpensive operation. Much of what happens in a system falls into this category, so you end up with a relatively speedy and snappy kernel implementation. For example, if two network devices are being bridged to transmit data from one network to another, and the network stacks and device drivers live in kernelspace, this operation can occur without any context change, saving time and increasing throughput.

Another benefit is that the userspace applications have a rich set of services which has a consistent interface available (usually the system call interface). This allows rich applications to be developed using a standard scheme.

Drawbacks

The same feature which is a benefit to monolithic kernels enabling speedy execution of kernelspace work, also is a drawback in some scenarios. If anything in kernelspace fails, it can potentially impact all of kernelspace. This is often addressed by having hardware enforced protections of kernelspace tasks as well as userspace tasks, which in many instances, ameliorates this issue.

Another drawback is that monolithic kernels must be updated atomically. They are typically tightly coupled and updates can be difficult to introduce.

Implications

If speed is a primary concern, monolithic kernels are good choices.

Microkernels

Microkernels are the minimalist brother to monolithic kernels. The essence of the microkernel can be summed up as:

Only the bare minimum to operate the kernel lives in kernelspace.

Nothing besides the minimal set of operations to consider the kernel operational resides in kernelspace in a microkernel. Many resources say this set is:

• Memory Management
• Process Management
• IPC

I like to add one more to the list because I feel it is a critical resource in a real system: Interrupt Management. Either way, it is easily seen that the microkernel contains fewer components running in kernelspace, hence the “micro” name.

The services that were once part of the kernelspace in the monolithic design are now run as “servers” in the userspace. These servers interact with the applications and the other servers via IPC, which is facilitated by the kernel. This of course has benefits, drawbacks and implications as well.

Benefits

One obvious benefit over the monolithic kernel is that the servers can be taken online/offline independently of the kernel and a failure in one does not induce a system-wide failure necessarily.

Drawbacks

Similarly to the monolithic kernel, the primary benefit to a microkernel also has a drawback associated. Because each server runs in userspace and relies on IPC as a primary mechanism of communications, constant context switching into and out of kernel mode means microkernels are typically slower than their monolithic counterparts.

Another drawback is that not all expected services are going to be available on a microkernel since they are effectively decoupled from kernel execution.

Implications

If service independence is most important, a microkernel is a good choice.

Key Takeaways

There are some aspects of the two primary kernel types I didn’t dig into, and I want to address them specifically below.

Security

There is a strong case for both monolithic and microkernels in the security perspective. In this realm, the truth is that both designs have strengths and weaknesses, and the kernel’s design is only one aspect of the security posture of the system.

Safety

Like security, safety has a place in both micro and monolithic kernels. The safety of each relies more on the implementation and usage of the particular kernel rather than its design.

Absoluteness of Design

In today’s software architectures, we regularly see hybrids of these two concepts where we can meld the benefits of each in a blended fashion. This is a bit of a hot topic because when the lines are blurred it is harder to define what falls into each category. For examples of hybridization, see this article.

Extension of the Concepts

By taking the concepts which drive the monolithic and particularly the microkernel design to extremes, we see some interesting outcomes. Take for instance the nanokernel. This is the extreme version of a microkernel, where additional services may not be available and the application must provide this all itself. This is tending towards being simply a hypervisor however, and may not be a microkernel at all. Another example is the exokernel where the hardware resources are basically un-abstracted completely, so the application must make all these decisions. Of note regarding exokernels, is that they have yet to be developed into a commercial-grade offering as of now, and are primarily a research concept.

The post Kernel Design: Microkernel vs. Monolithic appeared first on Jacob N Calvert.

Note: all the code, scripts, etc. are archived in a git repo at GitHub.

Diving In

As a refresher from the Types of Virtualization post, type-2 hypervisors are virtualization systems in which a Host OS sits between the Guest OS and the hardware. Sometimes, especially when the Guest OSes instruction set differs from the Host OSes instruction set, this type of hypervisors are called emulators. In this context, this type of hypervisor is well-suited for embedded systems development and test. For this deep dive, I will be focusing on the emulation side of type-2 hypervisors. (The other side of type-2 hypervisors are more like VirtualBox and are not particularly useful for embedded systems development).

QEMU as a Development Platform

QEMU is an open-source emulator which has broad community support and contains libraries for emulating a large number of targets of many different instruction sets and peripherals. This makes it a particularly useful tool for developing code for a target for which you don’t have real hardware.

Setting up a QEMU Target

Luckily, QEMU has prebuilt packages for most platforms you can simply install using the package manager of your choice. I’m going to be targeting the aarch64 (also referred to as arm64 in Linux land) architecture for emulating a virtual platform similar to the Raspberry Pi 3 (and newer). I have installed the package on my Linux Mint machine which provides qemu-system-aarch64. With this emulator available, I can invoke the emulator with the following command:

qemu-system-aarch64 -m virt -M 1G -kernel kernel.elf

This command will start up an ARMv8-A-based virtual machine, with 1GB of RAM, and load the code in kernel.elf.

The post Virtualization for Embedded Systems Series: Type-2 Hypervisors Deep Dive appeared first on Jacob N Calvert.

I recently stumbled upon HypriotOS while looking for Docker-ready distributions for my Raspberry Pi 3B+. I flashed this onto and SD card and started playing around with it. It works incredibly well, but I noticed that it was built for armv7l which is a 32-bit implementation. Since the Raspberry Pi 3B+ has a 4x core Cortex-A53 which is 64 bit, I wanted to make use of the 64 bit processor! I’ve worked with Yocto before (in fact, my day job uses Yocto), so I decided I’d build my own. For more on containers, see my other blog posts on what it is, what it can be used for, and how to use it.

Process

Prerequisites

Yocto is a build, not a distribution, so it is quite resource intensive. I would suggest having at least 60GB free disk space, and a quad core machine at a minimum. You’ll also need a Linux installation, and the build essentials installed (git, gcc, etc.).

EDIT: Checking the disk usage after the builds shows around 55GB used. On a 2C4T Intel Core i5 it took about 2H to build.

Project Setup

First, create a working directory and clone all the piece-parts we need to build.

mkdir rpi3bplus-build-yocto
cd rpi3bplus-build-yocto
git clone git://git.yoctoproject.org/poky
git clone git://git.openembedded.org/meta-openembedded
git clone git://git.yoctoproject.org/meta-raspberrypi
git clone git://git.yoctoproject.org/meta-virtualization 

Next create the project and add the base layers.

source poky/oe-init-build-env rpi64
bitbake-layers add-layer ../meta-raspberrypi
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-openembedded/meta-python
bitbake-layers add-layer ../meta-openembedded/meta-perl
bitbake-layers add-layer ../meta-openembedded/meta-networking
bitbake-layers add-layer ../meta-openembedded/meta-filesystems
bitbake-layers add-layer ../meta-virtualization

Edit the conf/local.conf file sections as needed:

MACHINE ??= "raspberrypi3-64"
CORE_IMAGE_EXTRA_INSTALL += "kernel-modules htop openssh iperf3 docker-ce bash ntp "

INHERIT += "extrausers"

EXTRA_USERS_PARAMS += " useradd pi; \
                       usermod  -p 'raspberry' pi; \
                       usermod  -a -G sudo pi; \
                       usermod -P root root; "

DISTRO_FEATURES_append = " virtualization"

Next we will build the image. This will take a while depending on how beefy your build machine is.

bitbake core-image-minimal
Parsing recipes: 100% |########################################################################################################################################################################################################| Time: 0:02:16
Parsing of 2540 .bb files complete (0 cached, 2540 parsed). 3827 targets, 140 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.44.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "raspberrypi3-64"
DISTRO               = "poky"
DISTRO_VERSION       = "3.0"
TUNE_FEATURES        = "aarch64 cortexa53 crc"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "master:6bb4a252199cfd5d44ad7ab6fc4118c80a1aae92"
meta-raspberrypi     = "master:a0a5d3848e76b7f90ef8e42c56211da78db1c7ba"
meta-oe              
meta-python          
meta-perl            
meta-networking      
meta-filesystems     = "master:d9f3e6dbed8e5d96f2069280f0a566af89afb2fa"
meta-virtualization  = "master:5fb77ae4c4e1015e40257f9e59e16c497e30c53c"

After a couple of hours, you will have a completed, ready to flash SD Image at <build>/rpi3bplus-build-yocto/rpi64/tmp/deploy/images/raspberrypi3-64/core-image-minimal-raspberrypi3-64.rpi-sdimg which you can burn to an SD card with:

dd if=./core-image-minimal-raspberrypi3-64.rpi-sdimg of=/dev/sdX status=progress

Your image should boot up and docker will be running!

NOTE: You will manually have to set the date to pull from Docker registries over HTTPS because the certification validation will fail otherwise. We could add and configure NTP to fix the but… for another day. Use:

date "+%d-%m-%C%y %H:%M:%S" -s "2019-12-22 00:30:01"

There is one last thing you’ll want to do. You’ll want to use GParted or similar tool to enlarge your filesystem partition to take up the rest of your SD card so you will have some room for container images. You need to do this because the default SD Image builder script only provides enough space for the root filesystem and no more. In my configuration it ended up being ~300MB, so I expanded it take up the remaining ~59GB. See example below:

Results

As it turns out, I have a system that can build and run 64-bit Docker images and is pretty darn slim. Here’s a screenshot of htop running.. only 12 processes – not bad!

Some Metrics

The is the minimal image and it has 12 processes, with an incredibly low footprint. I loaded up an Ubuntu container and installed iperf3, then ran a test to my laptop. I was impressed to find the following result:

[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.13  sec   116 MBytes  96.0 Mbits/sec    9             sender
[  5]   0.00-10.13  sec   113 MBytes  93.8 Mbits/sec                  receiver

Even running that test, the load on the Pi was:

root@raspberrypi3-64:~# cat /proc/loadavg 
0.05 0.22 0.19 1/141 2125
root@raspberrypi3-64:~# 

Downloads

The post Building a Customized Linux Image for Raspberry Pi with Yocto + Docker Support appeared first on Jacob N Calvert.

Note: all the code, Dockerfiles, etc. are archived in a git repo at GitHub.

Container History

Origins

A little history is needed before we jump into building and deploying containers. Docker, which is likely still the largest container technology provider by a longshot, was originally released in 2013. In just 6 years a huge community using containers sprang up around the concept – check out DockerHub for a sense of scale in the community. Preceding the Docker phenomenon, we had LXC (Linux Containers) which had been around since around 2008, but it really never had the sticking power that Docker has enjoyed. Fast-forward to today, and there are a handful of competing container technologies, all with similar features, but some have better ecosystems or support than others.

Where We Are Today

Recently there has a push for a standardization of the container frameworks that makeup these ecosystems. The Open Container Initiative is an open governance body for working towards standardization of the container framework. Many container systems have already adopted this and become OCI compliant or aligned. Docker has also bolstered the community by donating much of its container infrastructure components to the open source world for OCI to utilize.

Goal of the Open Container Initiative

I encourage you to go to the OCI’s website and read their mission documents in whole, but I’d like to provide the brief description here as it relates to our use of containers in embedded systems. The OCI intends to create a standard for representing containers and their basic runtime requirements and interfaces for portability. From their FAQs:

The mission of the Open Container Initiative (OCI) is to promote a set of common, minimal, open standards and specifications around container technology.

What is the mission of the OCI? FAQ (https://www.opencontainers.org/faq#faq1)

This is important to us in embedded engineering because we need this portability and compatibility to fully realize the use cases outlined in the last post. Since the OCI has not reached a sufficiently mature status, I will be sticking with Docker as the container ecosystem in this post.

Docker Containers: A Hands-On Example

Objectives

In the following sections we will accomplish the following:

1. Create a basic container image from a Dockerfile
2. Create a container to build and test a custom application
3. Create a container to host the custom application
4. Export the container
5. Run the container on a different platform

Creating a Basic Container

I’m starting with the assumption that Docker is installed and you’ve been able to run the Docker hello-world image. We will use a Dockerfile to compose our basic image. A Dockerfile is a script of sorts which instructs the engine to construct our container piece by piece. I’ve put a (very) basic Dockerfile below.

##########################################
# File: Dockerfile
# Author: Jacob Calvert <jcalvert@jacobncalvert.com>
# Date: Nov-08-2019
# 
# This is a basic Dockerfile 
#
##########################################

# start from the basic busybox image
FROM busybox

# put a file in our container
COPY hello-world.txt .

What I have in my workspace on my host machine is as follows:

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ ls
Dockerfile  hello-world.txt
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ cat hello-world.txt
hello, world!

To build the Docker image from the Dockerfile, you run a ‘docker build’ command. The ‘-t’ specifies a tag by which we’ll reference this image, and the ‘.’ specifies the directory for the Dockerfile – in this case, the current directory.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker build -t basic .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM busybox
 ---> 020584afccce
Step 2/2 : COPY hello-world.txt .
 ---> 11e3eeda8f8e
Successfully built 11e3eeda8f8e
Successfully tagged basic:latest
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ 

Let’s view and run our image now.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker image ls
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
basic                             latest              11e3eeda8f8e        42 seconds ago      1.22MB
ubuntu                            latest              775349758637        8 days ago          64.2MB
busybox                           latest              020584afccce        9 days ago          1.22MB
hello-world                       latest              f2a91732366c        23 months ago       1.85kB
quantumobject/docker-zoneminder   latest              469615ab191d        24 months ago       1.15GB
mysql/mysql-server                latest              a3ee341faefb        2 years ago         246MB
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker run -it basic
/ # ls
bin              dev              etc              hello-world.txt  home             proc             root             sys              tmp              usr              var
/ # cat hello-world.txt 
hello, world!
/ # exit
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ 

What are we looking at here? First we list our available local images with the docker image ls command. We can see that I have several images on my development machine, including basic which was created only a few moments ago from our build command. Next I run the image we created with docker run -it <image>. The -it flags are for –interactive and –tty. These two together essentially present the container’s console as a pseudo-tty device in your terminal, in interactive mode. It is as if you are sitting in front of another machine.

Next, you can see that I have a different prompt. This is the container’s prompt. I type ls and we can see our hello-world.txt is there on the filesystem of our container as we desired via the COPY command in the Dockerfile. Lastly, we type exit which ends the busybox process and exits the container.

Now let’s look at the persistent state parts of Docker.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker system info
Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 6
 Server Version: 19.03.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.4.0-165-generic
 Operating System: Linux Mint 18
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.55GiB
 Name: jacob-aspire-mint
 ID: Z2GE:5Y2A:K4LP:UC4O:QV3A:4JMR:5RLW:2DHQ:WANN:2RA3:VKJ2:UZMI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Using docker system info we can see that there is 1 container and that it is stopped (not running). How do we see what that container is?

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
c76e0de1a530        basic               "sh"                4 minutes ago       Exited (0) 3 minutes ago                       silly_kalam

We can use the ps command to see information about our containers. Notice the STATUS column. Our container’s execution has exited. This begs the question: can we restart a container and pick up where we left off? The answer of course is yes!

jacob@jacob-aspire-mint /media/jacob/jacob/Documents/Workspaces/Blog/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker container start -i silly_kalam 
/ # ls
bin              dev              etc              hello-world.txt  home             proc             root             sys              tmp              usr              var
/ # 

Notice the command difference this time around. I am using docker container start -i <name> where the name is a name given to the container at run time. The container can be referenced by its human-friendly name as I have done here, or by its ID (the long UUID). Also note the -i flag; this is to open it as an interactive session again. No -t is needed since the TTY device has already been allocated. So can we work inside this running container like a real machine and see the state persist? Indeed we can as well! Let’s see it in action.

/ # cd home/
/home # mkdir -p user/workspace/
/home # cd user/workspace/
/home/user/workspace # echo "another file!" > another.file
/home/user/workspace # ls
another.file
/home/user/workspace # exit

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker container start -i silly_kalam 

/ # ls
bin              dev              etc              hello-world.txt  home             proc             root             sys              tmp              usr              var
/ # cd home/user/workspace/
/home/user/workspace # ls
another.file
/home/user/workspace # cat another.file 
another file!
/home/user/workspace # 

It may be a little hard to follow, but here’s what has happened. From inside our previously started container named silly_kalam, I have created a directory /home/user/workspace. Next, I created a file named another.file and filled it with “another file!”. I then exited the container. Next, I started the container again, changed directory to my created directory, and printed out the contents of another.file.

Using this example we can see that the content inside a container is not purely ephemeral, but can persist over many starts/stops. For more persistent storage check out Docker’s Volume subsystem.

Creating a Development Environment Container

Using the knowledge gained through building a basic container, we will now create a container to use as a development environment for our example application.

The Application Specs

We want to build a simple application which demonstrates the portability of containers and also does something we can test. We also want it to be a simple application for demo purposes. With this in mind, our application will be a simple data modem. It will take in data from one source medium and spit it out as another medium. Below is a list of basic requirements codified for this application:

• Translate data from a serial device to UDP
  • Serial will be at 115200/8N1 configuration
  • UDP will listen on a configurable port
  • TTY device will be configurable
• Modem will log messages to container console

To satisfy these requirements we need to be able to build application code for the container. Why use a container to build applications? Why not just use your host machine’s development environment? Repeatability is the answer. Rather than struggle to maintain a development environment across multiple developers who have their own flavors of Linux distro and preferences and so on, we can distribute a “builder” container which has all the dependencies and tools needed to build the application (albeit a simple one in this case) from scratch and is completely independent of the host it runs on.

Building and Using the Container

For our “builder” container we create the following Dockerfile:

##########################################
# File: Dockerfile
# Author: Jacob Calvert <jcalvert@jacobncalvert.com>
# Date: Nov-09-2019
# 
# This Dockerfile creates a development environment for 
# building a sample application
#
##########################################

# start from the basic ubuntu image
FROM ubuntu

# install the needed tools in our container
RUN apt-get update && apt-get install build-essential git net-tools -y

This Dockerfile starts with an Ubuntu base image, and adds the three common packages build-essential, git, and net-tools. I won’t show the entire build process but at the end you should be able to run the new container as we did in the basic use case.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ docker run -it dev-env
root@32984d05f73f:/# ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:02  txqueuelen 0  (Ethernet)
        RX packets 19  bytes 2853 (2.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@32984d05f73f:/# gcc
gcc: fatal error: no input files
compilation terminated.
root@32984d05f73f:/# 

Now in another tab let’s copy our source into our builder image (note: we could have simply git cloned it as well, but in many areas I work in, network access is a no-go).

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ docker cp modem.c jovial_rhodes:/

And now we can see this file in our running container.

root@32984d05f73f:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  modem.c  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@32984d05f73f:/# 

So let’s build our application, and grab the resulting binary.

root@32984d05f73f:/# gcc modem.c -o modem -lpthread
root@32984d05f73f:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  modem  modem.c  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@32984d05f73f:/# 

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ docker cp jovial_rhodes:/modem ./modem
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ ls
Dockerfile  modem  modem.c
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ 

We have successfully used a container to build an image for deployment. The application is a simple one, but it illustrates the point of using a “builder” container for repeatable builds. Now we can move on to deploying our application. Of course there’d be rigorous testing in a real-world application, but we can neglect that for the purposes of demonstration.

Creating a Deployment Environment Container

We want to now create a container which will start our application on boot, and run that application until we exit the container. We will start with our basic Ubuntu, and add a few things as shown below.

##########################################
# File: Dockerfile
# Author: Jacob Calvert <jcalvert@jacobncalvert.com>
# Date: Nov-09-2019
# 
# This Dockerfile creates a deployment environment for 
# the sample application
#
##########################################

# start from the basic ubuntu image
FROM ubuntu

# create a app/bin directory in our container
RUN mkdir -p /app/bin 

# copy in the app and a startup script
COPY modem /app/bin
COPY start-modem.sh /

# set the entry point
ENTRYPOINT /start-modem.sh

Notice a new directive here? The ENTRYPOINT directive is what will be run on startup of the container. Now let’s take a look at the contents of that script.

#!/bin/sh
/app/bin/modem -d $TTY_DEVICE -p $PORT

Simple right? All it does is start our application with some environment variables as parameters.

Testing our Deployment Container

Here’s where the magic of containers will really start to shine.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/deploy-env $ sudo docker run -it -e TTY_DEVICE=/dev/ttyACM0 -e PORT=9000 --device=/dev/ttyACM0 deploy-env 

This time we will run our container image with a few extra parameters. First, the -e flag will inject key=value pairs as environment variables. That’s how our script will know what those are at run time. Next we pass through the /dev/ttyACM0 device from the host machine to the docker container. If we modify that line to –device=/dev/ttyACM0:<custom container path> we can give the serial device a specific path in the container, otherwise it just gets identity mapped. Docker sets up a default NAT network on the host on the 172.17.0.0/24 subnet, so we will have access to this container at an IP address in that range once we start it. Also, the device hanging on /dev/ttyACM0 is just printing out a sequence for test data every second.

So what’s it look like when it runs? From the container you see:

 
Device selected is '/dev/ttyACM0'
Port base selected is 9000
UDP TX: '3
'
UDP TX: '0
'
UDP TX: '1
'
UDP TX: '2
'
UDP TX: '3
'
UDP TX: '4
'
UDP TX: '5
'
UDP TX: '6
'
UDP RX: 'hello world!
'
UDP TX: '7
'
UDP TX: '8
'
UDP TX: '9
'
UDP TX: '10
'
UDP TX: '11
'
UDP TX: '12
'
UDP TX: '13
'
UDP RX: 'hello virtualization!
'
UDP TX: '14
'
UDP TX: '15
'
UDP TX: '16
'
UDP TX: '17
'
UDP TX: '18
'
UDP TX: '19
'
UDP TX: '20
'
UDP TX: '21
'
UDP TX: '22
'
'DP TX: '23
UDP TX: '
'
UDP TX: '24

And from the netcat (nc) session on my host:

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ nc -u 172.17.0.2 9000
hello world!
hello virtualization!
14
15
16
17
18
19
20
21
22
23
24
^C

I’ve typed the “hello” statements into the netcat session and it is receiving the increment sequence from the serial device via UDP on the remote end. We have successfully created and deployed a containerized application!

Exporting the Containerized Application

Since we now have a complete application all self-contained in a container, we can export this to run on other systems. In Docker, this is as simple as:

docker save -o deployment-environment.tar deploy-env:latest

We now have a TAR archive of all the layers building up our container. This is a portable format you can move around to deploy on different machines by importing it onto another host and running it.

Running the Containerized Application on Another Machine

I used scp to copy my TAR file to another machine and imported it into Docker using:

jacob@dev-ubuntu:~# docker load < deployment-environment.tar 
bd59016c97ec: Loading layer [==================================================>]  2.048kB/2.048kB
8c558935f47c: Loading layer [==================================================>]   16.9kB/16.9kB
0cc408f84946: Loading layer [==================================================>]  2.048kB/2.048kB
Loaded image: deploy-env:latest
jacob@dev-ubuntu:~# docker image ls
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
deploy-env                latest              d427611b8747        About an hour ago   64.2MB

Now we can run the app on a different machine just like our original host machine. (I omitted any parameters, I just want to see the modem binary print out errors).

jacob@dev-ubuntu:~# docker run -it deploy-env 
Device selected is '-p'
Failed to open device '-p'

So what’s the point?

So what’s the point? Couldn’t we just copy our modem binary to the other machine and run it in the native host? Sure, for this application, because it has no special dependencies. But imagine if your application depends on QT5, TensorFlow, and a pre-trained Data Model? Making sure all that stuff gets installed on the target host machine is a nightmare. Having a single TAR file with all dependencies built in, with your application properly parameterized ready to launch makes portability and usabilty a much easier sell, especially for embedded systems.

Said another way: if the only requirement to upgrade your application or add additional applications to a fielded embedded system is that you have the right container infrastructure, it is much easier to rapidly update existing capabilities and to deploy new capabilities to the edge with a high confidence of success.

Wrapping Up

In this post, we focused on a practical example of using containers. It’s easy to see how this technology has the ability to be incredibly impactful for embedded systems. In the next post, we will take a look at type-2 hypervisors and how to use them for embedded systems.

I hope you enjoyed the content of this post! If you did, feel free to comment or shoot me a message over at the contact page!

The post Virtualization for Embedded Systems Series: Containers Deep Dive appeared first on Jacob N Calvert.

Containerization

Simplified Management

One of the low hanging fruits of container virtualization is the ability to manage your application images in a straightforward and simplified manner. Because container images carry with them all the dependencies needed to run the application the container is intended for, we eliminate the need to prepare the host environment to deploy that application. In other words, we no longer need to worry if our application is deployed on six system variants with six different hardware configurations running six different host OS revisions. We simply package our app for the container infrastructure, and ship it as a container. Simple.

Rapid and Widespread Deployments

Utilizing the same image management capabilities, we can easily deploy our applications on a multitude of platforms, since we do not need to worry about the underlying host OS too much. For example, you have written an application which provides critical situational-awareness (SA) functionality to the Soldier. Your application simply needs the SA data as an input and it serves a web page with the resulting SA data displayed for fast, efficient dissemination to the necessary parties. This application packaged as a traditional binary would require configuration management to be performed on any deployment platform, and care would need to be taken each time the application was updated to manage dependencies. If deployed as a containerized solution, the container carries its dependencies along with it. We can then deploy in real-time to platforms we may not have directly designed the application for use on, and provide valuable capabilities to the warfighter in a blink.

Type-2 Hypervised Systems

Scratching the Nostalgia Itch

Who remembers the 80’s? Not me! But I did have a Sega as a kid. Gaming platforms these days have become more PC-like than embedded system, but in the not so distant past, gaming systems were 100% embedded systems by design. But what about scratching that itch for Mortal Combat on the Sega in 2019 without searching E-bay endlessly? This is a perfect fit for a type-2 hypervisor to help you out. The RetroPie Project is all about bringing different emulated platforms into the modern age on a Raspberry Pi. Via the hypervised environment (and assuming you have legal rights to the console title you’re emulating) you can spin up a virtual Sega and load up a copy of Mortal Combat to throw down with the boys on a Saturday night. I call that solving problems with technology!

Cross-Platform Development

Frequently in the embedded systems world, we find ourselves developing code on one platform to be deployed on another. Type-2 hypervisors make it easy to test our developed code on something that mimics the real target platform with a high level of fidelity. Using a tool like QEMU, you can quickly deploy ARMv7 code on top of a dual-core Cortex-A9 VM while being hosted on your Ubuntu Linux running an Intel chip. In the same vein, cross platform development can be achieved with a type-2 hypervisor like VirtualBox as well, by running a VM of your deployment platform and developing directly “on-platform.” For a highly flexible and incredibly powerful commercial platform that supports the system simulation paradigm for developing, deploying, testing and more for embedded systems and beyond, check out Wind River’s Simics.

Type-1 Hypervised Systems

An aside: Type-1 hypervisors are, in my opinion, one of the most useful virtualization strategies for embedded systems. For starters, embedded devices often have less processing power and RAM than their datacenter counterparts, so the low overhead factor for type-1 hypervisors makes them ideal for squeezing as much performance as possible out of the hardware platform.

Application Consolidation

Functionality implemented by software has historically been included in embedded systems in a federated manner. This has been especially true for safety critical functions in software. Federated systems have been the standard model for safety critical systems since software began to take a major role in system operation.

In federated systems, each function often had its own “box” in the system. This means for every function or function group there was an entire dedicated hardware and software stack to support it. This model was also reinforced due to limited computing resource availability on single core processors. When a single core chip was maxed out on processing power, the only solution was to add another “box.” Each function belonging to an individual “box” obviously increases the size, weight, and power (SWaP) required for the system.

By consolidating the various applications into one hardware platform we reduce the SWaP required for the system-of-systems, and to do this we can use a type-1 hypervisor. We can accomplish even more if we utilize a multicore processor with a type-1 hypervisor. This application consolidation mechanism is easily seen in the selection of an ARINC653-capable OS for the Boeing 7E7 Common Core System.

Security Risk Mitigation

One of the glaring issues of fielding embedded systems is that often, once fielded, the option to update them does not exist. When embedded systems are expected to work 24/7/365 from the time of production until replacement, this “long tail” needs to be considered. Whenever security holes are found in a fielded embedded system, it can be difficult to design a mitigation that can be applied without rearchitecting the entire system. Take for example the following scenario: a safety-critical embedded device has been deployed with an custom home-grown RTOS. It was recently discovered that this home-grown code has a major security flaw in the TCP/IP stack, despite the fact that the application only uses the serial interface. The RTOS is inflexible and it would cause too much rework to update the RTOS and application and get an ATO again. Instead of modifying the RTOS and application, you simply run a type-1 hypervisor underneath the RTOS and application, and use the type-1 hypervisor to restrict which hardware can be accessed by the RTOS and application. By simply virtualizing the application, we’ve moved it “up the stack” so that we can better manage the resources it has access to, and which resources have access to it – in this way, we can patch a large number of security issues simply by using a VM.

Legacy Migration

Since embedded systems often have long lived deployments as identified in the Security Risk Mitigation example, the addition of new features and the migration of legacy code is also a difficult task many times. Again, having a type-1 hypervisor to move your application “up the stack,” can ease the difficulty in this migration. For example, you deployed an application on Linux and it has been fielded for seven years. Recently you have new requirements for the application which requires some functionality to be moved to an RTOS to be responsive and keep up with demand. Rather than making a sweeping change to port all functionality to an RTOS, using a type-1 hypervisor can allow you to retain most of the functionality in Linux in one VM and port just the necessary functions over to an RTOS in another VM. All this while still supporting the same hardware platform.

More Examples

Here are a few more examples that don’t necessarily fall under the umbrella of embedded systems, but help round out the picture.

Scalable Services by Design

Previously I noted that Docker is a popular container engine or daemon for managing containers on a given host machine. I also mentioned Kubernetes as an orchestration engine to deploy and manage these containers hosted by Docker. Pairing these two technologies, we are able to design services that are scalable in nature right from the start.

But what exactly is meant by scalable services though? Imagine the following example. You’ve created a web application and are using Docker for the various services that make up your application. You start with less than 1000 daily users and are spinning up additional containers manually to manage the load. Suddenly, your app is featured on Slashdot and you begin experiencing the Slashdot Effect. Your running containers are overloaded and your users are denied access – not good for a fledgling web app! Now, consider if your application’s containers were being managed and monitored by Kubernetes. The same Slashdot Effect begins to occur, but the Kubernetes engine reacts by spinning up additional containers to handle the spike in load. No crash for your app, and tons of happy new users. Thanks to this type of virtualization, you can be infinitely flexible to support your users’ demand, on-demand.

Network Function Virtualization

Network function virtualization as a concept has grown in popularity rather rapidly over the past several years. It straddles the line between IT-grade virtualization and embedded systems because it takes what would have traditionally been a dedicated embedded device performing one function or group of functions and has virtualized it so that it can sit as a virtual appliance or virtual machine on enterprise grade computing platforms. This is typically deployed as a hypervised environment with management and orchestration software running on top to handle the chaining of functions. For more info on NFV, check out the Wikipedia article.

Computing Power On Demand

My website is hosted by DigitalOcean on a Droplet. A Droplet is DigitalOceans’ term for a Virtual Private Server (VPS). They have many physical behemoth servers in datacenters all over the world, and using virtualization, are able to divvy up the resources on those servers into virtual machines called Droplets. I can log into my Droplet as if it is a physical machine and do with it as I please. What’s really neat about DigitalOcean (and many other providers like it) is that they provide a way to spin up these VPS units on demand via a web API. We could actually use this API to spin up a swarm of computing power long enough for it to crunch some data for us and then turn them back off, releasing the hardware to another user! In fact, DigitalOcean uses QEMU and libvirt to accomplish their virtualization and provide this on demand compute to consumers like me. Pretty cool!

Wrapping Up

In this post, we looked at how the various types of virtualization technology can be applied in a variety of ways. In the next couple of posts, I’ll be taking a look at software platforms that enable these virtualization mechanisms, and building up example systems with mostly FOSS pieces to show what things we can accomplish and to demonstrate some of the concepts I noted in this post.

The post Virtualization for Embedded Systems Series: Applications in the Real World appeared first on Jacob N Calvert.

If you didn’t catch the intro to this series, you can read a little about the motivation for these posts in the previous post.

Types of Virtualization Technology

When considering the delineations between different types of virtualization technology, it helps to draw the line based on where the Virtual Machine (VM) or container touches the hardware. Let’s look at this aspect of virtualization first.

Containerization

Containerization is the application of virtualization to create lightweight, small footprint “services” that sit on top of a host operating system. In the previous post, I linked to Kubernetes which is an orchestration engine (among other things) for containers. Containers are often used to segregate applications or services into their own user space for a multitude of reasons. One of the primary reasons is scalability.

Using an orchestrator like Kubernetes, you can write your applications’ services such that they can dynamically scale up or down to meet demand by spinning up or down containers “in the cloud.” In fact, this is the exact model that many SaaS (software as a service) providers are using to maximize their capabilities on a limited set of hardware resources. Check out Amazon AWS for more details on this model (it’s pretty neat!). This model typically requires the host OS to have support for this container creation and can typically only support containers of the same flavor as the host OS (this is not always true, but there are caveats)^[1]. For example with Docker as your container manager, you can only host Linux containers on a Linux host OS.

At first glance, containerization may not seem like a good fit for embedded systems, but when we explore use cases specifically for embedded devices, we’ll find there are some pretty neat applications of the technology in various industries.

Process Virtual Machines

Process Virtual Machines (PVMs) are very similar to containerization, but instead of presenting a full user space for your application, you’re only allowed one process. Due to the similarities in structure to containers, I’m going to lump these two together.

Type 2 Hypervisors

This type of virtualization is related to containerization in the respect that a host operating system sits between the virtualized application and the hardware. It is also quite a bit different, as there is also an entire guest operating system sitting between your application and the hardware.

This gives you more isolation in some respects, but can also create performance issues. One major benefit of type 2 hypervisors is that you can run mixed varieties of guest OSes independent of your host OS. VirtualBox is a good example of this type of virtualization. Using VirtualBox, I can run a Linux Mint guest and a Windows 7 guest on my Windows 10 host, side-by-side. Another benefit of type 2 hypervisors is that you can also run mixed architectures on the same host platform. For example, using QEMU, I can run a PowerPC-based Linux image on my Intel x86_64 processor. In this mode however, the performance is greatly impacted (although there have been many improvements), and you will not get the same performance as running on the native instruction set of the host OS.

Type 1 Hypervisors

Type 1 hypervisors are the flavor of virtualization which brings the guest OS closest to the hardware, and provides the best performance in a hypervised environment. In this type of virtualization, the hypervisor itself is the host OS, and provides all critical facilities to the guest OS with minimal impact to the guest OSes execution.

Another critical difference here is that the guest OSes must share the same instruction set architecture (ISA). In other words, if your host platform is an ARMv8 cluster, your guest OSes must all run ARMv8 machine code, however, some architectures have support to run 32bit and 64bit side by side on the same chip depending on configuration (we’ll explore this later.)

Other Considerations

With all the types of virtualization technology I’ve listed, there are a few other considerations they have in common which are worth at least noting.

Processor Oversubscription

Processor oversubscription is the concept of assigning more virtual CPU cores to your guest OS machines than you have physical CPU cores on your hardware. Some environments allow you to do this, and it is not a universal feature. It’s also not universally desired for every virtualization solution. Typically, your physical CPU’s cores are not utilized at 100%. Allowing oversubscription so you can make better use of the processing cycles makes sense, but what happens if a malicious application starts hogging all of the cycles on one core? Depending on the virtualization technology and its configuration, the other guest OSes may suffer as a result.

RAM and Storage Oversubscription

This concept is similar to Processor Oversubscription, but applies to the physical RAM and the non-volatile storage in your host hardware. Again, there is a chance that a guest OS will need to use its RAM resource, and because it is oversubscribed it may not be available, leading to processing delays, or at worst, failure. Similarly, perhaps a critical log message needs to be written to a disk, but the storage quota for the disk is already met due to oversubscription – a failure is likely!

Hardware-assisted Hardware Sharing

Sounds redundant doesn’t it? It’s not, I promise! This concept is a hardware-based capability and varies by platform. The basic concept is that a device that multiple guest OSes may want to use (like an Ethernet controller) has some built-in capabilities to help each VM or guest OS use the shared resource, while keeping their data separate. An example of this is SR-IOV. Single Root I/O Virtualization is a technology related to PCIe where PCIe resources can be divvied up for use among several guest OSes. Check out this Microsoft article for more information on SR-IOV.

Wrapping Up

Now that we’ve explored several different flavors of virtualization technology, I’d like to zero in on just the ones that we are likely to use in embedded systems – containers and hypervisors. In the next post, we will walk through some examples of how each of these can be applied to embedded systems, and what benefits they can bring.

References

[1] Docker: Linux Containers on Windows Hosts ( https://www.docker.com/blog/preview-linux-containers-on-windows/ )

The post Virtualization for Embedded Systems Series: Types of Virtualization appeared first on Jacob N Calvert.

In today’s technology driven world, virtualization plays a giant role in our day-to-day life. Services like Amazon AWS and DigitalOcean are ubiquitous as service providers to all sorts of companies. These provide the backbone to our daily browsing and Internet usage habits. Entire ecosystems have been built around this concept as well. Take Kubernetes for example^[2]. Kubernetes is a suite of software designed from the ground up to support deploying, managing, balancing, monitoring and more for distributed containers across dissimilar architectures and platforms. Similarly, VMWare’s vCenter Orchestrator^[3] performs similar functions for fully hypervised guest OSes or Virtual Machine (VMs). Typically we use virtualization to make the best use of hardware resources. This has been particularly beneficial in the realm of web services and cloud computing since the advent of powerful multicore processors backed by massive swathes of RAM. Now that multicore processors are no longer only in the data center, and they have become sufficiently small and efficient enough to be used in embedded systems, is virtualization the next step for embedded devices? Some may say we’re already there (looking at you Android Runtime^[4]). I say that virtualization is only going to become more prevalent in embedded systems as we consolidate more functionality into smaller footprints.

I’m starting this series on Virtualization for Embedded Systems to share my knowledge and explore the state of the art for embedded device virtualization. I’ll be taking a look at how various virtualization mechanisms work, which architectures work well for virtualization, software platforms that make virtualization easy and attainable, and the benefits of going virtualized.

I hope to cover these aspects in three or so articles, so stay tuned!

References

[1] Wikipedia contributors. (2019, September 23). CP/CMS. In Wikipedia, The Free Encyclopedia. Retrieved 16:34, October 31, 2019, from https://en.wikipedia.org/w/index.php?title=CP/CMS&oldid=917270626

[2] Kubernetes.io (2019, October 31). Homepage. Retrieved 18:55, October 31, 2019, from https://kubernetes.io/

[3] blog.vmware.com (2014, June 27). VMware vSphere Blog. Retrieved 19:02, October 31, 2019, from https://blogs.vmware.com/vsphere/2014/06/getting-started-vmware-vcenter-orchestrator-building-many-virtual-machines.html

[4] Wikipedia contributors. (2019, October 2). Android Runtime. In Wikipedia, The Free Encyclopedia. Retrieved 19:59, October 31, 2019, from https://en.wikipedia.org/w/index.php?title=Android_Runtime&oldid=919289272

The post Virtualization for Embedded Systems Series: Introduction appeared first on Jacob N Calvert.