The External Connector

The mini DIN connector on the back has a fairly standard pinout as shown below. Note it says 4800bps max for the DATA IN. I want to find out why!

In the service manual, I start by locating pin 1 on the DIN connector.

Internal Circuitry

Next I trace this to a pair of 2-1 muxes.

These muxes are operated together (that is, their control signal is the same signal). From my investigation, this is the mux pair that is activated when selecting the TNC operation for the radio. The right-most mux, IC114, selects between MIC IN audio from the mic port, through a series of filters, or the TNC port we’re interested in. The left-most mux, IC112, selects which TX generator chip we’re going to be using. The path I’ve highlighted in red is the TNC audio input path I’m trying to trace.

Transmission Modulation

Next the signal travels to the MICIN pin on a chip with designator IC109, and part number BK4811B.

This is where the magic of modulation happens, and where I ultimately found the reason for only supporting up to 4800bps. Scouring the web for a datasheet for this chip, I found a PDF I could download. Checking the relevant sections of the chip’s documentation, I found the culprit, screenshot below:

The audio path into the chip is shown in the blocks at the bottom. The bullet points 6 & 7 tell the details of why 9600bps isn’t usable. The low-pass and high-pass blocks cannot be bypassed on this chip, and the maximum achievable width is ~2.7k, which is not enough for 9600bps. You can turn off the compander and pre-emphasis, but there’s no way to expand the LPF->HPF network bandwidhth. To run 1200-4800bps you really only need ~1kHz, but 9600bps needs to have much more.

Wrapping Up

Now that I am satisfied that the radio can *not* in fact TX 9600bps (but can receive it, perhaps another analysis in another post to come), I will get on with getting DireWolf setup for APRS, and FlDigi for other fun digital modes.

The post Why the Alinco DR-735T can’t transmit at 9600bps appeared first on Jacob N Calvert.

Note: all the code, scripts, etc. are archived in a git repo at GitHub.

Diving In

As a refresher from the Types of Virtualization post, type-2 hypervisors are virtualization systems in which a Host OS sits between the Guest OS and the hardware. Sometimes, especially when the Guest OSes instruction set differs from the Host OSes instruction set, this type of hypervisors are called emulators. In this context, this type of hypervisor is well-suited for embedded systems development and test. For this deep dive, I will be focusing on the emulation side of type-2 hypervisors. (The other side of type-2 hypervisors are more like VirtualBox and are not particularly useful for embedded systems development).

QEMU as a Development Platform

QEMU is an open-source emulator which has broad community support and contains libraries for emulating a large number of targets of many different instruction sets and peripherals. This makes it a particularly useful tool for developing code for a target for which you don’t have real hardware.

Setting up a QEMU Target

Luckily, QEMU has prebuilt packages for most platforms you can simply install using the package manager of your choice. I’m going to be targeting the aarch64 (also referred to as arm64 in Linux land) architecture for emulating a virtual platform similar to the Raspberry Pi 3 (and newer). I have installed the package on my Linux Mint machine which provides qemu-system-aarch64. With this emulator available, I can invoke the emulator with the following command:

qemu-system-aarch64 -m virt -M 1G -kernel kernel.elf

This command will start up an ARMv8-A-based virtual machine, with 1GB of RAM, and load the code in kernel.elf.

The post Virtualization for Embedded Systems Series: Type-2 Hypervisors Deep Dive appeared first on Jacob N Calvert.

Here’s a breakdown: on the FT232R, the max speed I can set the adapter to is 3MHz. This has given me a transfer speed (loading via GDB) of around 3KB/s. Not too bad for small projects of only a few hundred KiB. But now my code size is approaching a ~5 MiB, the debug cycle was way too long… 5MiB @ 3KB/s = ~28mins to load… not good!

Still in my pursuit of a cheap JTAG debugger (I mean come on! It’s just a serial protocol!), I did buy something. The Adafruit FT232H breakout is exactly what I was looking for. At only $15 with Amazon Prime shipping, I was back to debugging at a reasonable speed in two days.

The FT232H breakout is about as barebones as it gets. The board breaks out untouched ACBUS0-ACBUS7 and ADBUS0-ADBUS7 to 0.10″ pitch headers. Since the FT232H has a single MPSSE channel, I can use this breakout to get faster JTAG speeds.

My configuration files from the previous post had to change a little to accommodate. I’ll break it down below in comments in the file because it’s a little bit cryptic.

#
# FTDI USB Hi-Speed to MPSSE Breakout from Adafruit
#
# This should work for any bare FT232H
#

# Setup driver type
adapter driver ftdi

# 30000 kHZ -> 30MHz
adapter speed 30000

# Using JTAG (also could be SWD)
transport select jtag

# Common PID for FT232H
ftdi_vid_pid 0x0403 0x6014

# Set sampling to allow higher clock speed
ftdi_tdo_sample_edge falling


# Layout
# On this breakout, the LEDs are on ACBUS8 and ACBUS9, can't assign them
# registers are <ACVALUE><ADVALUE> <ACCONFIG><ADCONFIG>
# so we set 0x0308 to mean only ACBUS nTRST and nSRST, ADBUS3 (TMS) asserted high
# and we set 0x000B to mean only AC3,AC2,AC0 outputs -> (TMS,TD0, TCK)
ftdi_layout_init 0x0308 0x000b

# Pins
# pin name      | func. |
# --------------|-------|
# ADBUS0        | TCK   |
# ADBUS1        | TDI   |
# ADBUS2        | TDO   |
# ADBUS3        | TMS   |
# ACBUS0        | nTRST |
# ACBUS1        | nSRST |
#---------------|-------|

# When data == oe -> pins are switched from output to input to give
# the tri state (L, H, Hi-Z) effect 
ftdi_layout_signal nTRST -data 0x0100 -oe 0x0100
ftdi_layout_signal nSRST -data 0x0200 -oe 0x0200

So now it is as simple as connecting the corresponding pins on this breakout to the RPi3, and running OpenOCD again.

With the increased throughput of this MPSSE-supported interface, I now get ~587KB/s. Much better! It only takes ~8s to load my image now.

The post Better JTAG on the Cheap with the FT232H appeared first on Jacob N Calvert.

Note: all the code, Dockerfiles, etc. are archived in a git repo at GitHub.

Container History

Origins

A little history is needed before we jump into building and deploying containers. Docker, which is likely still the largest container technology provider by a longshot, was originally released in 2013. In just 6 years a huge community using containers sprang up around the concept – check out DockerHub for a sense of scale in the community. Preceding the Docker phenomenon, we had LXC (Linux Containers) which had been around since around 2008, but it really never had the sticking power that Docker has enjoyed. Fast-forward to today, and there are a handful of competing container technologies, all with similar features, but some have better ecosystems or support than others.

Where We Are Today

Recently there has a push for a standardization of the container frameworks that makeup these ecosystems. The Open Container Initiative is an open governance body for working towards standardization of the container framework. Many container systems have already adopted this and become OCI compliant or aligned. Docker has also bolstered the community by donating much of its container infrastructure components to the open source world for OCI to utilize.

Goal of the Open Container Initiative

I encourage you to go to the OCI’s website and read their mission documents in whole, but I’d like to provide the brief description here as it relates to our use of containers in embedded systems. The OCI intends to create a standard for representing containers and their basic runtime requirements and interfaces for portability. From their FAQs:

The mission of the Open Container Initiative (OCI) is to promote a set of common, minimal, open standards and specifications around container technology.

What is the mission of the OCI? FAQ (https://www.opencontainers.org/faq#faq1)

This is important to us in embedded engineering because we need this portability and compatibility to fully realize the use cases outlined in the last post. Since the OCI has not reached a sufficiently mature status, I will be sticking with Docker as the container ecosystem in this post.

Docker Containers: A Hands-On Example

Objectives

In the following sections we will accomplish the following:

1. Create a basic container image from a Dockerfile
2. Create a container to build and test a custom application
3. Create a container to host the custom application
4. Export the container
5. Run the container on a different platform

Creating a Basic Container

I’m starting with the assumption that Docker is installed and you’ve been able to run the Docker hello-world image. We will use a Dockerfile to compose our basic image. A Dockerfile is a script of sorts which instructs the engine to construct our container piece by piece. I’ve put a (very) basic Dockerfile below.

##########################################
# File: Dockerfile
# Author: Jacob Calvert <jcalvert@jacobncalvert.com>
# Date: Nov-08-2019
# 
# This is a basic Dockerfile 
#
##########################################

# start from the basic busybox image
FROM busybox

# put a file in our container
COPY hello-world.txt .

What I have in my workspace on my host machine is as follows:

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ ls
Dockerfile  hello-world.txt
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ cat hello-world.txt
hello, world!

To build the Docker image from the Dockerfile, you run a ‘docker build’ command. The ‘-t’ specifies a tag by which we’ll reference this image, and the ‘.’ specifies the directory for the Dockerfile – in this case, the current directory.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker build -t basic .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM busybox
 ---> 020584afccce
Step 2/2 : COPY hello-world.txt .
 ---> 11e3eeda8f8e
Successfully built 11e3eeda8f8e
Successfully tagged basic:latest
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ 

Let’s view and run our image now.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker image ls
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
basic                             latest              11e3eeda8f8e        42 seconds ago      1.22MB
ubuntu                            latest              775349758637        8 days ago          64.2MB
busybox                           latest              020584afccce        9 days ago          1.22MB
hello-world                       latest              f2a91732366c        23 months ago       1.85kB
quantumobject/docker-zoneminder   latest              469615ab191d        24 months ago       1.15GB
mysql/mysql-server                latest              a3ee341faefb        2 years ago         246MB
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker run -it basic
/ # ls
bin              dev              etc              hello-world.txt  home             proc             root             sys              tmp              usr              var
/ # cat hello-world.txt 
hello, world!
/ # exit
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ 

What are we looking at here? First we list our available local images with the docker image ls command. We can see that I have several images on my development machine, including basic which was created only a few moments ago from our build command. Next I run the image we created with docker run -it <image>. The -it flags are for –interactive and –tty. These two together essentially present the container’s console as a pseudo-tty device in your terminal, in interactive mode. It is as if you are sitting in front of another machine.

Next, you can see that I have a different prompt. This is the container’s prompt. I type ls and we can see our hello-world.txt is there on the filesystem of our container as we desired via the COPY command in the Dockerfile. Lastly, we type exit which ends the busybox process and exits the container.

Now let’s look at the persistent state parts of Docker.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker system info
Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 6
 Server Version: 19.03.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.4.0-165-generic
 Operating System: Linux Mint 18
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.55GiB
 Name: jacob-aspire-mint
 ID: Z2GE:5Y2A:K4LP:UC4O:QV3A:4JMR:5RLW:2DHQ:WANN:2RA3:VKJ2:UZMI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Using docker system info we can see that there is 1 container and that it is stopped (not running). How do we see what that container is?

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
c76e0de1a530        basic               "sh"                4 minutes ago       Exited (0) 3 minutes ago                       silly_kalam

We can use the ps command to see information about our containers. Notice the STATUS column. Our container’s execution has exited. This begs the question: can we restart a container and pick up where we left off? The answer of course is yes!

jacob@jacob-aspire-mint /media/jacob/jacob/Documents/Workspaces/Blog/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker container start -i silly_kalam 
/ # ls
bin              dev              etc              hello-world.txt  home             proc             root             sys              tmp              usr              var
/ # 

Notice the command difference this time around. I am using docker container start -i <name> where the name is a name given to the container at run time. The container can be referenced by its human-friendly name as I have done here, or by its ID (the long UUID). Also note the -i flag; this is to open it as an interactive session again. No -t is needed since the TTY device has already been allocated. So can we work inside this running container like a real machine and see the state persist? Indeed we can as well! Let’s see it in action.

/ # cd home/
/home # mkdir -p user/workspace/
/home # cd user/workspace/
/home/user/workspace # echo "another file!" > another.file
/home/user/workspace # ls
another.file
/home/user/workspace # exit

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/basic $ docker container start -i silly_kalam 

/ # ls
bin              dev              etc              hello-world.txt  home             proc             root             sys              tmp              usr              var
/ # cd home/user/workspace/
/home/user/workspace # ls
another.file
/home/user/workspace # cat another.file 
another file!
/home/user/workspace # 

It may be a little hard to follow, but here’s what has happened. From inside our previously started container named silly_kalam, I have created a directory /home/user/workspace. Next, I created a file named another.file and filled it with “another file!”. I then exited the container. Next, I started the container again, changed directory to my created directory, and printed out the contents of another.file.

Using this example we can see that the content inside a container is not purely ephemeral, but can persist over many starts/stops. For more persistent storage check out Docker’s Volume subsystem.

Creating a Development Environment Container

Using the knowledge gained through building a basic container, we will now create a container to use as a development environment for our example application.

The Application Specs

We want to build a simple application which demonstrates the portability of containers and also does something we can test. We also want it to be a simple application for demo purposes. With this in mind, our application will be a simple data modem. It will take in data from one source medium and spit it out as another medium. Below is a list of basic requirements codified for this application:

• Translate data from a serial device to UDP
  • Serial will be at 115200/8N1 configuration
  • UDP will listen on a configurable port
  • TTY device will be configurable
• Modem will log messages to container console

To satisfy these requirements we need to be able to build application code for the container. Why use a container to build applications? Why not just use your host machine’s development environment? Repeatability is the answer. Rather than struggle to maintain a development environment across multiple developers who have their own flavors of Linux distro and preferences and so on, we can distribute a “builder” container which has all the dependencies and tools needed to build the application (albeit a simple one in this case) from scratch and is completely independent of the host it runs on.

Building and Using the Container

For our “builder” container we create the following Dockerfile:

##########################################
# File: Dockerfile
# Author: Jacob Calvert <jcalvert@jacobncalvert.com>
# Date: Nov-09-2019
# 
# This Dockerfile creates a development environment for 
# building a sample application
#
##########################################

# start from the basic ubuntu image
FROM ubuntu

# install the needed tools in our container
RUN apt-get update && apt-get install build-essential git net-tools -y

This Dockerfile starts with an Ubuntu base image, and adds the three common packages build-essential, git, and net-tools. I won’t show the entire build process but at the end you should be able to run the new container as we did in the basic use case.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ docker run -it dev-env
root@32984d05f73f:/# ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:02  txqueuelen 0  (Ethernet)
        RX packets 19  bytes 2853 (2.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@32984d05f73f:/# gcc
gcc: fatal error: no input files
compilation terminated.
root@32984d05f73f:/# 

Now in another tab let’s copy our source into our builder image (note: we could have simply git cloned it as well, but in many areas I work in, network access is a no-go).

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ docker cp modem.c jovial_rhodes:/

And now we can see this file in our running container.

root@32984d05f73f:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  modem.c  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@32984d05f73f:/# 

So let’s build our application, and grab the resulting binary.

root@32984d05f73f:/# gcc modem.c -o modem -lpthread
root@32984d05f73f:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  modem  modem.c  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@32984d05f73f:/# 

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ docker cp jovial_rhodes:/modem ./modem
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ ls
Dockerfile  modem  modem.c
jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ 

We have successfully used a container to build an image for deployment. The application is a simple one, but it illustrates the point of using a “builder” container for repeatable builds. Now we can move on to deploying our application. Of course there’d be rigorous testing in a real-world application, but we can neglect that for the purposes of demonstration.

Creating a Deployment Environment Container

We want to now create a container which will start our application on boot, and run that application until we exit the container. We will start with our basic Ubuntu, and add a few things as shown below.

##########################################
# File: Dockerfile
# Author: Jacob Calvert <jcalvert@jacobncalvert.com>
# Date: Nov-09-2019
# 
# This Dockerfile creates a deployment environment for 
# the sample application
#
##########################################

# start from the basic ubuntu image
FROM ubuntu

# create a app/bin directory in our container
RUN mkdir -p /app/bin 

# copy in the app and a startup script
COPY modem /app/bin
COPY start-modem.sh /

# set the entry point
ENTRYPOINT /start-modem.sh

Notice a new directive here? The ENTRYPOINT directive is what will be run on startup of the container. Now let’s take a look at the contents of that script.

#!/bin/sh
/app/bin/modem -d $TTY_DEVICE -p $PORT

Simple right? All it does is start our application with some environment variables as parameters.

Testing our Deployment Container

Here’s where the magic of containers will really start to shine.

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/deploy-env $ sudo docker run -it -e TTY_DEVICE=/dev/ttyACM0 -e PORT=9000 --device=/dev/ttyACM0 deploy-env 

This time we will run our container image with a few extra parameters. First, the -e flag will inject key=value pairs as environment variables. That’s how our script will know what those are at run time. Next we pass through the /dev/ttyACM0 device from the host machine to the docker container. If we modify that line to –device=/dev/ttyACM0:<custom container path> we can give the serial device a specific path in the container, otherwise it just gets identity mapped. Docker sets up a default NAT network on the host on the 172.17.0.0/24 subnet, so we will have access to this container at an IP address in that range once we start it. Also, the device hanging on /dev/ttyACM0 is just printing out a sequence for test data every second.

So what’s it look like when it runs? From the container you see:

 
Device selected is '/dev/ttyACM0'
Port base selected is 9000
UDP TX: '3
'
UDP TX: '0
'
UDP TX: '1
'
UDP TX: '2
'
UDP TX: '3
'
UDP TX: '4
'
UDP TX: '5
'
UDP TX: '6
'
UDP RX: 'hello world!
'
UDP TX: '7
'
UDP TX: '8
'
UDP TX: '9
'
UDP TX: '10
'
UDP TX: '11
'
UDP TX: '12
'
UDP TX: '13
'
UDP RX: 'hello virtualization!
'
UDP TX: '14
'
UDP TX: '15
'
UDP TX: '16
'
UDP TX: '17
'
UDP TX: '18
'
UDP TX: '19
'
UDP TX: '20
'
UDP TX: '21
'
UDP TX: '22
'
'DP TX: '23
UDP TX: '
'
UDP TX: '24

And from the netcat (nc) session on my host:

jacob@jacob-aspire-mint /workspace/virtualization-for-embedded-systems-series/containers-deep-dive/dev-env $ nc -u 172.17.0.2 9000
hello world!
hello virtualization!
14
15
16
17
18
19
20
21
22
23
24
^C

I’ve typed the “hello” statements into the netcat session and it is receiving the increment sequence from the serial device via UDP on the remote end. We have successfully created and deployed a containerized application!

Exporting the Containerized Application

Since we now have a complete application all self-contained in a container, we can export this to run on other systems. In Docker, this is as simple as:

docker save -o deployment-environment.tar deploy-env:latest

We now have a TAR archive of all the layers building up our container. This is a portable format you can move around to deploy on different machines by importing it onto another host and running it.

Running the Containerized Application on Another Machine

I used scp to copy my TAR file to another machine and imported it into Docker using:

jacob@dev-ubuntu:~# docker load < deployment-environment.tar 
bd59016c97ec: Loading layer [==================================================>]  2.048kB/2.048kB
8c558935f47c: Loading layer [==================================================>]   16.9kB/16.9kB
0cc408f84946: Loading layer [==================================================>]  2.048kB/2.048kB
Loaded image: deploy-env:latest
jacob@dev-ubuntu:~# docker image ls
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
deploy-env                latest              d427611b8747        About an hour ago   64.2MB

Now we can run the app on a different machine just like our original host machine. (I omitted any parameters, I just want to see the modem binary print out errors).

jacob@dev-ubuntu:~# docker run -it deploy-env 
Device selected is '-p'
Failed to open device '-p'

So what’s the point?

So what’s the point? Couldn’t we just copy our modem binary to the other machine and run it in the native host? Sure, for this application, because it has no special dependencies. But imagine if your application depends on QT5, TensorFlow, and a pre-trained Data Model? Making sure all that stuff gets installed on the target host machine is a nightmare. Having a single TAR file with all dependencies built in, with your application properly parameterized ready to launch makes portability and usabilty a much easier sell, especially for embedded systems.

Said another way: if the only requirement to upgrade your application or add additional applications to a fielded embedded system is that you have the right container infrastructure, it is much easier to rapidly update existing capabilities and to deploy new capabilities to the edge with a high confidence of success.

Wrapping Up

In this post, we focused on a practical example of using containers. It’s easy to see how this technology has the ability to be incredibly impactful for embedded systems. In the next post, we will take a look at type-2 hypervisors and how to use them for embedded systems.

I hope you enjoyed the content of this post! If you did, feel free to comment or shoot me a message over at the contact page!

The post Virtualization for Embedded Systems Series: Containers Deep Dive appeared first on Jacob N Calvert.

Everything, and I mean everything is “smart” these days. Everyone has heard of the Internet of Things, and we are living through the emergence of some incredibly revolutionary connectivity. Some pretty cool and useful concepts have come out of it, admittedly, but there’s also some drawbacks.

I really enjoy the idea of having smart outlets that I can command with my voice, and there are outlets-a-plenty on Amazon, but can I trust them? I’ve decided to use my engineering skills to put a one of these outlets to the test.

Project Plan

Test Subject

I purchased a pack of Etekcity Smart Plugs which claim to be Alexa-enabled, and feature energy monitoring.

Test Plan

I want to test many aspects about the smart plug, so I laid out a test plan of tests I want to run.

• Record the network traffic coming and going from the device (couldn’t complete, see below)
• Pop open the device, see what hardware is under the hood
• Write some custom firmware for the device

Project Execution

Preparation

These devices generally work in the following way:

1. Connect to the device while it’s acting as an Access Point (AP)
2. Configure the device via a proprietary app
3. Device transitions from AP to Station Mode and connects to your local network
4. User connects Alexa/Google/etc services to device

When I started to look at the app for these devices, I decided against installing and using it per the user documentation. Here’s why.

The app for setting up the plug required the following permissions:

• Camera
• Contacts
• Location
• Call logs
• Phone
• Storage
• way, way more.

See the screenshot below for the full list.

So right off the bat, I decided to not install this app, simply because it had way, way too many permission oddities.

Opening the Device Up

I unpackaged these units and immediately noticed that there’s nary a screw anywhere. Upon closer inspection, you could see a thin line where the unit was clamshelled together. I grabbed my tools (a skinny flat-head screwdriver and a 3D printer sharpened spatula) and started poking for the release tabs.

After prodding the device for a few moments, I heard the distinctive “pop” of a plastic catch tab unclasping. I was able to raise up the edge of the clamshell and began working my way around the unit. Within seconds, the unit was open and the guts were exposed.

Jackpot!

The first thing I noticed was that the WiFi module is labeled as an ESP01! Very fortuitous for me since there is so much open source goodness out there the ESP8266 and ESP8285. Next up is figuring out how to flash this device. Others I’ve looked at hid the traces for the TX, RX, and GPIO0 on the backside, so after the module was soldered on, it was pretty much inaccessible. Not so, with this guy! There were test points on the PCB, perfectly labeled for everything I needed: TX, RX, GPIO0, 5V and GND.

I tacked on some temporary wires as shown in the photo and connected to my laptop via a USB-TTL Serial cable. My cable is a 3.3V logic cable with 5V on the VCC wire, so I powered the chip from my cable as well.

I opened picocom and set the baud rate to 115200, and voila! I had terminal output. Regrettably, I didn’t capture the terminal output, but it mentioned some version numbers, the ESP SDK version, and had the classic boot up text from the ESP01.

On the less fortuitous side, I couldn’t see any other chips (like the one used for energy monitoring), so I suppose they are on the opposite side of the PCB, which is molded into the shell.

Flashing the Device

I had all the right inputs, but would it flash? The answer was yes! I fired up the Arduino IDE and wrote a simple Hello, World app, and performed the flashing procedure: pull GPIO0 low, apply power, let GPIO0 float, begin flashing.

Without any fuss and to my surprise, the unit flashed without delay. I now had a working smart plug unit that I could flash very easily! I wrote up some custom firmware that allowed over-the-air (OTA) updates and removed my temp wires.

Custom Firmware

After a few iterations of writing “probing” apps and loading it onto the device, I figured out that the pinout from the ESP01 was the following (irrelavent pins ignored):

Of these pins, I knew how to use most right away. It was those energy monitoring pins I couldn’t figure out. Since I didn’t know how to understand the output, and I didn’t know what IC was generating the output, I couldn’t really make use of it. I noted a few characteristics about the signals however:

1. The signal on P12 stayed constant in time. I measured the low and high pulse duration, and they were the same (within 1 uS).
2. The signal on P13 was constant low when the relay was inactive, and toggled when there was a device connected and the relay was active.

From these observations, I made the assumption that P12 was the voltage pin, and P13 must be current or power. I also made the assumption that the signal must be a 50% DC signal which varies in frequency only. Using a pin change interrupt I was able to verify this assumption.

Now that I understood the signal profile, I needed to understand how to convert it to something usable. I had to do a lot of Googling around, but I stumbled upon a forum which showed many of these cheap WiFi outlets use a Chinese chip HLW8012. There’s only a Chinese language datasheet available for the HLW8012 on the web, but again, a little Googling revealed this guy’s website which is in English! He had a ton of useful information on his website, and it corroborated the signals I was observing, so I again made the assumption that this is the right chip.

Using the formulas outline on the last website I found, I was able to turn the P12 signal pulse width into a voltage, and verify that it was pretty darn close using my Kill-a-Watt. ( I highly recommend these little devices, they are super useful if you like to tinker). I got values that were ± 0.75% and so I concluded this was the right chip. Next was determining if the P13 pulse was a power or current. Since it stopped with the opening of the relay and started with the closing of the relay, it could have been both. From the previous website, the poster noted that the HL8012 chip had two outputs, CF and CF1. One of the inputs (SEL) made one output switch between voltage and current, and the other was always power. There was my answer! If the output on P12 was constant regardless of load, and I’d been able to get a voltage from the readings which was fairly accurate, then the other has to be power. Converting the P13 pulse to power also jived with my Kill-a-Watt’s readings (although with a little worse accuracy, it is ± 3% by my best estimation).

First Release

When I had identified how to operate all the parts of this little device, I set about writing a firmware load which enabled all the uses I desired. I ended up with the following feature-set in my first release:

• Alexa integration via Phillips Hue emulation
• Over-the-Air updates for touch-free updating
• Web Interface
  • Device Control
  • Device debug information
  • Device Energy Monitoring graphs
• LED indication for device state
• Button interface for manual on/off

Wrapping Up

I only accomplished 2 of my 3 stated goals, but I did have a ton of fun doing it! At the end of the day, I’ve got two functional smart outlets with energy monitoring which have custom firmware. By virtue of using custom firmware, I can control what is needed to control the device as well, so no shady apps with too many permissions. I liked these so much, and they were so easy to customize, I have decided to ordered 4 more for around the house!

The post Hacking a Smart Outlet appeared first on Jacob N Calvert.

Terms and Definitions

Right up front I want to define the terms I’ll be using throughout the post for clarification’s sake. A core is a single processing element, irrespective of the cache, RAM, peripherals, etc. which could be connected to it. A processor is a chip consisting of 1 to N cores. When talking about an application on a core of a multiprocessor, I mean any piece of code. That could be a simple Hello World, to a complex bare-metal piece of software, to a full-up OS.

Asymmetric Multiprocessing (AMP) Operation

One of the most straightforward ways to utilize more cores on the same processor is to treat each core as an individual single core processor all to its own. This mode is called asymmetric multiprocessing or AMP. This mode of operation allows you to run N applications on the N cores and treat them like N separate processors. One of the key advantages of this mode is that you can run different applications on each core, giving you greater flexibility as to your use of the multicore chip. There are some caveats to this mode of operation however. Most multicore processors do not include N copies of the supporting hardware units to make the multicore processor operate as N truly independent processors. Take for example, the NXP QorIQ P4080. It has 8 cores, but only 2 DRAM controllers. So even running in an AMP mode, you must share the DRAM controllers among the 8 cores. This couples the cores together so that they are not truly independent. This is a typical configuration for multicore systems wherein N cores share memory controller and data paths to peripherals and can present some planning challenges when using an AMP mode of operation.

Symmetric Multiprocessing (SMP) Operation

This is the mode of operation most people think of when envisioning a multicore processor. This mode of operation combines all the cores into a set of co-equal processing elements. The application is responsible for assigning work to each of the cores in the set and all hardware belongs to or is owned by the application. This is how most OSes work on multicore processors. Take for example Linux. Linux manages the workload spread across the N cores via the scheduler. The scheduler decides what process’s threads are ready to run, and divvies them up across the cores available. The Linux kernel also handles which threads of execution can access which hardware pieces, but this is a discussion for another day! The big benefit to SMP is that you have a “supervisor application” (usually the OS) managing all the processing resources and the software developer doesn’t have to put much thought into which core his software will run on. However, this abstraction can also introduce some latency into the system when a cache flush is required to move a thread of execution from one core to another.

Simultaneous Multithreading (SMT) Operation

This mode of operation is less well understood in general than its precursor SMP, although the concept is the same in operation. SMT applies to processors which support higher levels of utilization of the hardware units on the chip and requires a little bit of understanding of how a processor works. The short and sweet is that in any given processor, there are some common elements such as an ALU, instruction fetch unit, instruction decode unit, memory interface components, etc. When a processor is executing a given instruction, not all of those elements are active at the same time, even with a pipelined design. SMT allows software to view the single core in a multicore system as itself having more than one processing element available. In the parlance this is said as having a “N cores, M threads”. From the software’s point of view, after SMT is enabled, the processor simply has M cores. Most Intel processors support SMT and plenty of others do as well, like the NXP QorIQ T2080.

Bound Multiprocessing (BMP) Operation

This is a concept closely related to SMP in theory again, however with a little bit of a twist. In a BMP system, a single application owns the whole set of cores, but the application can decide to bind certain threads of execution to certain cores rather than floating them around the cores as needed. If a system supports SMP, it can easily be modified to support BMP as well (and most already have this baked in, see processor affinity settings in your favorite OS). The major benefit of BMP over SMP is that you don’t have threads of execution floating from one core to another, and so you have tighter-grained control over the execution. You do however lose some throughput capability as you may end up waiting on a bound thread to run on another core before your software can do its work, and so careful thought must be given to this mode of operation.

Mixed Multiprocessing Operation

This is not an officially recognized term, but it’s one that I think fits the bill. There are some systems that will allow you to use both AMP and SMP in the same processor. For example, when using a hypervised environment like Wind River’s Helix Virtualization Platform, you may sometimes want a configuration like the following:

When a system supports both AMP and SMP in the same chip complex, I consider this a mixed multiprocessing environment. For more information on virtualization and how it can achieve these mixed modes operation, check out my series on Virtualization in Embedded Systems.

Going Further

I hope this gave a quick intro to the different modes a multicore system can be in, but if you want more detail still, see this excellent article from NXP on these modes. It leaves out SMT, but covers AMP, SMP, and BMP pretty thoroughly.

The post Multicore Processor Modes of Operation appeared first on Jacob N Calvert.

The world of computing as we know it today only began in the last century or so, and really only since the invention of the bipolar transistor. Fast-forward to the modern era of computing wherein we have unfathomable computing power surrounding us daily. Computing platforms and the related technology is driving advancement in every industry across the globe. From research to reality, computing platforms have been a benefactor to society time and time again. The largest benefit of modern computing platforms is that they do the heavy lifting for us – the complex calculations, the ugly mathematics, the trial and error search for a solution – without tying up a real human-being’s time and energy. Simply put, a computing platform generally is the most useful and powerful when it requires the least human interaction to complete its tasks. The ability to monitor and manage these computer platforms is as important as the jobs they perform. Additionally, as computing platforms pack more power and capability into tighter size, weight, and power constrained units, the need to easily monitor and manage these high-powered machines becomes increasingly evident.

Exploration in the Defense Industry

The Defense Industry has wholeheartedly embraced the concept of COTS (commercial off-the-shelf) tech insertion for speeding up the development and deployment of advanced capabilities to protect and defend the national interest. The wide use of COTS VPX and VME backplane technology in the Defense Industry has driven the need for a dependable platform management solution. Since many of these computing platforms are deployed in rugged and harsh operating environments, it is desirable and necessary to monitor the health of each unit in a system and manage fail-over situations should they arise. Operators are not always co-located to these systems and are not able to physically monitor and mitigate these issues. This implies that a platform management solution should be robust enough to manage fail-over situations, remotely, with limited human interaction.

High Level Solution

In keeping with the theme of COTS solutions for Defense Industry problems, the natural progression for implementing a platform management solution is to use the COTS product already in use by other industries. The Intelligent Platform Management Interface (IPMI), is the industry standard for platform management. The IPMI architecture “defines standardized, abstracted interfaces to the platform management subsystems” [1]. The IPMI specification describes a message based request/response protocol in which commands are grouped by functional sets using their Network Function Code or NetFn for short.

This protocol can be initiated over a wide variety of interfaces such as a serial port, a Keyboard Controller Style (KCS) interface, or an Intelligent Platform Management Bus (IPMB). The serial port, KCS interface and other interfaces comprise a class of interfaces known as System Interfaces. System Interfaces are used by the unit itself to communicate with its onboard IPMI controller. The IPMB belongs to a class of interfaces which are used to communicate with a unit remotely; i.e., not from within an application level software on the unit itself.

These commands over an IPMB are used to monitor and manage the individual units in a larger system. In IPMI terms, each unit would be called a Field Replaceable Unit or FRU. An example of a FRU might be one Single Board Computer (SBC) in a VPX chassis. Network Function Codes exist for managing sensor data, reading and writing FRU data, configuring and acknowledging events and alerts, powering on or off a FRU, and more.

The IPMI spec also allows for extensions to its standard yet robust set of commands. One extension is the VITA Standards Organization’s VITA46.11 Specification [2]. The VITA46.11 spec implements a set of states and commands on top of the IPMI specification which allow for more granular control of the FRU by defining FRU states, as well as a set of standard logical sensors which summarize the FRU health.

A device implementing these standards on a FRU is called an Intelligent Platform Management Controller or IPMC. Baseboard Management Controller (BMC) is another term used to describe an IPMC. The VITA46.11 spec divides implementations into Tier I implementations and Tier II categories. A Tier I IPMC implements a subset of the standard, whereas a Tier II IPMC implements the entirety of the standard.

Pairing the IPMI standard commands with VITA46.11 additions, an IPMC can be used to monitor and manage a FRU in a wide variety of situations. The real value in having an IPMC onboard a FRU, is that when multiple FRUs exists in a chassis together, an IPMI compliant Chassis Manager has access to an incredible wealth of information about what is occurring in the system. For example, in a system with multiple FRUs which contain IPMC v2.0 compliant IPMCs, the Chassis Manager can instruct each FRU to monitor certain voltages or temperatures and send an alert known as a Platform Event Message if an out-of-range event occurs on the FRU. Not only can the FRU be instructed to monitor itself, but the Chassis Manager can request any and all sensor data at any time from any FRU. This is a valuable asset in the view of system management.

Conclusion

The need to monitor and manage computing platforms grows with the increasing complexity and power of those platforms. In the Defense Industry, COTS products are a solution to quick and standards compliant tech insertion with respect to computing platforms, and platform management is no different. Using standards compliant IPMCs will reduce or remove the need for custom platform management solutions and ease system management greatly.

References

[1] IPMI v2.0

[2] VITA 46.11 

The post Better Systems Management Through COTS Technology appeared first on Jacob N Calvert.