Setting up MiNiFi CPP 0.2 Telematics Tracking

Frequently in our IoT edge use-cases there is a need to transmit GPS data over varying types of data links such as cellular (3G/4G) and WiFi. This post will review how to setup MiNiFi CPP on a Raspberry Pi Zero-W with GPS, 4G Cellular Modem, and WiFi.  Alot of the focus* here will be how you can use the MiNiFi product today to enable your SandBox platform which  may result in this example feeling more ‘clunky’  to enable alot of actions the tracking use-case required this early on with MiNiFi.

*How to setup the Cloud/DataCenter NiFi servers  for Site-2-Site is not discussed here. 

WARN – The version of MiNiFi CPP used is 0.2; this is very early for this product and if your reading this Blog Post from a historical perspective it may be useful but its highly likely that if a year has past the product has matured more and would change most of the following designs and user experience for the better. 

Prototype Dongle Config

Edge & Platform Requirements

Many people ask me for my opinion on if the ‘Edge’ is part of the Spoke in a Hub & Spoke architecture. Given the increase in Edge computing I think of the edge as the Tire, or the Rubber on the road. I add this new layer into the traditional Hub & Spoke design because its an independent device that comes and goes based on jagged connectivity.

  • Edge (TIRE) Requirements
    • Captures DataPoints @ Given Sensor Intervals
      • GPS, Cell Strength, etc
      • Can be configured as N-Seconds or CRON like scheduling
    • Back Pressure
      • Can Prioritize what expires when storage becomes full
      • Can Prioritize which messages in Queues/Connections to send downstream first
    • Multiple Data Links (Cellular and WiFi)
      • Links can be Encrypted (TLS)
    • Delivery Guarantees
    • Can Receive From Hubs (*In MiNiFi C2)
    • Security
      • The Hub and Edge both authenticate each-other with certificates
    • Very Small Resource FootPrint of ‘Framework Software [MiNiFi]’
      • See Prior Blog Post graphs at the Bottom for MiNiFi on Pi Zero-W with Idle System Metrics and even the workflow discussed here.
  • Platform (HUB & SPOKE) Requirements
    • Delivery Guarantees
    • Back Pressure
      • If a DataLink is down for days, keep the important data
    • Transfer Throughput
      • Transfers should perform AUTOMATIC Load Balancing, and discovery between nodes
        • No Reverse Proxy
        • No Load Balancer (Soft or Hard)
          • Cluster AutoDiscover for Hosts to Load Balance on
      • Compression
        • Reduce egress costs
        • Improve throughput (if spare cpu cycles exist)
    • Security
      • Edge, Spoke, and Hub all use certificates to authenticate
      • Hub can PULL data from Spoke
        • Prevents exposing the DataCenter (No Inbound Connections)
        • Can use a HTTP_PROXY
          • This makes many security folks happy to have a single audit point and is valid so long as the bandwidth is available
      • Data Links can be Encrypted

Logical Architecture of the Platform

From an Architecture point of view each of these ‘zones’s is its own Architecture Failure Domain. This allows each of the Zones to fail without causing a outright failure to any of its connected systems. In the most simple of aspects the above Platform is utilizing a Hub & Spoke (and Tire) architecture where the DataCenter itself is the Hub, Cloud the Spoke… There are many reasons to go to this design; Security Surface reduction for multiple utility solutions needing data flow,  and provides a bi-directional pipeline between the Cloud and DataCenter with Site-2-Site (See NiFi C2 for MiNiFi BiDirectional Comms), Regional Cloud for Latency, etc.  Important to note that Site-2-Site capabilities in NiFi provides native load-balancing between clusters, compression, encryption, delivery guarantees for data, and if required can connect through an HTTP_PROXY to access clusters on other network segments.

Hardware

  • Edge Device
    • Pi Zero-W (using a U3 C10 Flash Storage Card)
    • BU-353-S4 GPS Dongle
    • Huawei Boltz 4G LTE Modem Unlocked
      • Using Google Fi SIM
    • Micro USB 2.0 Male to Female OTG Cable
    • Anker USB Hub
  • Cloud
    • 4GB 2 Core Server
  • DataCenter
    • 8 Core 96GB Ram Server

Software

  • Edge Device
    • Raspbian GNU/Linux 8 (jessie)
    • MiNiFi CPP 0.2 (built from source)
    • GPSD
    • usb-modeswitch
    • libqmi
    • Bash
  • Cloud
    • Ubuntu 16.04 Server
    • NiFi 1.3
  • DataCenter
    • Ubuntu 16.04 Server
    • NiFi 1.3

Code Flow and Implementation

The data-flow designed here is very basic with the goal of just getting GPS collected while in the field. Four Shell Scripts were built to provide the needed functionality and all directly link to the Site-2-Site Remote Process Group. All connections are configured as LIFO so that the most recent data is sent to the cloud whenever we regain connectivity and then the older data will get set when it can. Each connection was configured based on the important of the data so that GPS datapoints have up to 100MB of storage while simpler items like the WiFi connection status will only gather 10MB worth of events before expiring them. Additionally time expiration is being used; GPS will wait 168 hours before expiring events it has not sent, while other informational events expire every 48 hours.

GPS Tracking Flow in NiFi

Shell Scripts in ExecuteProcess Processors

If you have a desire to replicate the above flow you can find all there scripts and templates linked below. The scripts all generate JSON details about the actions they perform such as SCANNING for WiFi SSIDs, Resetting the entire 4G LTE Modem, Disk Metrics and even the GPS. The GPS JSON is generated by the GPSD application itself so we just capture it as is.

Setting It all Up

Headless Pi Configuration

First image a new flashcard with the correct version of Raspbian.

To enable ssh by default drop a 0-byte file called ‘ssh’ into ‘/boot/’ this will enable ssh which on later versions of Raspbian is disabled by default for security.

touch /path/to/fashcard/boot/ssh

Configure to AutoJoin your WiFi SSID by editing /etc/wpa_supplicant_wpa_supplicant.conf, this is different the the WiFi script as this will occur at boot even if the scripts are not running. You will also have to configure your /etc/network/interface  to autoconnect.

vi /etc/wpa_supplicant_wpa_supplicant.conf
#
network={
   ssid="YourNetworkSSID"
   psk="Your Network's Passphrase"
   key_mgmt=WPA-PSK
}
#

vi /etc/network/interfaces
# You may want to use dhcp and not manual, your device may not be named wlan0
allow-hotplug wlan0
iface wlan0 inet manual
 wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
#

Install the GPS Daemon and have NTP sync time from GPS Satellites so time is kept even without a Data Link.

#Install Software
sudo apt-get install gpsd gpsd-clients python-gps ntp

#Setup GPSD Defaults
#sudo vi /etc/default/gpsd
#
START_DAEMON="true"
GPSD_OPTIONS="n"
DEVICES="/dev/ttyUSB0"
USBAUTO="false"
GPSD_SOCET="/var/run/gpsd.sock"
#
#Test GPS 
sudo service gpsd restart 
cgps -s 

#Setup NTP to Sync from GPS Time 
sudo vi /etc/ntp.conf
#
# GPS Serial data reference
server 127.127.28.0 minpoll 4 maxpoll 4
fudge 127.127.28.0 refid GPS
fudge 127.127.28.0 time1 +0.140 # coarse processing delay offset
# GPS PPS reference
server 127.127.22.0 minpoll 4 maxpoll 4 prefer
fudge 127.127.22.0 refid PPS
fudge 127.127.22.0 flag3 1 # enable kernel PLL/FLL clock discipline
#

#Wait 5 Minutes and check if GPS is setting time 
sudo service gpsd restart 
sudo service ntp restart 
ntpq -p
NTP Using GPS for Time

Setup the 4G LTE Modem, this seems complex but isn’t to bad. If you Google you will find lots of other methods to do this including WVDail but here QMI will be used as it appears to be the currently most common method. Your device may hipup and require you to run modeswitch again. See the LTE-Connect script for a way to brute force handle problems.

#Install QMI, ppp, and usb_modeswitch

apt-get install libqmi-utils ppp usb-modeswitch

#Setup your Cell APN, below is configured for Google Fi. You will have to find the APN your cellular service uses.

echo "APN=h2g2" >/etc/qmi-network.conf

#Find the Huawei Modem Vendor and Product Info
 lsusb | grep Huawei

Bus 001 Device 004: ID 12d1:1505 Huawei Technologies Co., Ltd. E398 LTE/UMTS/GSM Modem/Networkcard

#If your using the same 4G modem as I am during the install its going to show up as 12d1:1505 but later it will become 12d1:1506 after using USB_SWITCH, this is expected.

cd /etc/usb_modeswitch.d/

tar -xzvf /usr/share/usb_modeswitch/configPack.tar.gz 12d1\:1505

vi 12d1\:1505
#
DefaultVendor= 0x12d1
DefaultProduct=0x1505
TargetVendor= 0x12d1
TargetProductList="140b,1506,150f"
MessageContent="55534243123456780000000000000011062000000100000000000000000000"
#
sudo usb_modeswitch -c /etc/usb_modeswitch.d/12d1\:1505

sudo reboot

#At each reboot use below to manually connect to WAN and renew your DHCPs

sudo qmi-network /dev/cdc-wdm0 start
sudo dhclient -r /dev/cdc-wdm0
sudo dhclient/dev/cdc-wdm0
sudo qmi-network /dev/cdc-wdm0 status

#wait a few minutes and check for an IP, then test the interface
ifconfig
ping -I wwan0 www.google.com

#you can also use traceroute -i $interface www.google.com

 It Lives!

Depending on the chipset where MiNiFi CPP is installed you may be required to compile it from source code (as was the case for the Raspberry Pi as its ARM.) If your testing this on an x86 system you can just download the tarball. The tarball folder absolute path becomes $MINIFI_HOME. Because the scripts in the ExecuteProcess Processors take care of starting most everything the services need such as GPS Daemons init.d scripts are not required for everything OTHER THAN the MiNiFi service itself; you can install MiNiFi for auto-start:   $MINIFI_HOME/bin/minifi.sh install

https://nifi.apache.org/minifi/download.html

#RHEL/CentOS:
yum install -y epel-release
yum install -y leveldb

#Debian/Ubuntu:
apt install -y libleveldb-dev
apt install -y libxml2

#Reuse Existing Template and Make sure to update the Site-2-Site Host and Script locations $MINIFI_HOME/conf/config.yml 
wget https://pastebin.com/raw/E52LvF2b -O $MINIFI_HOME/conf/config.yml

vi $MINIFI_HOME/conf/config.yml
# Update Host and Port of Site-2-Site
Remote Processing Groups:
- name: http://r01s01n01:8080/nifi
  url: http://r01s01n01:8080/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  Input Ports:
  - id: 089c253b-015d-1000-2077-c6fa9242c2b9
    name: rx
    comment: ''
    max concurrent tasks: 1
    use compression: false
    Properties:
        Port: 12551
        Host Name: r01s01n01
#


#By default all the scripts are located at '/iot/scripts/'
export scriptsdir=/iot/scripts mkdir -p $scriptsdir
wget https://pastebin.com/raw/RcdyQGTJ -O $scriptsdir/disk-report.sh
wget https://pastebin.com/raw/n4NiXnE9 -O $scriptsdir/gps-logger.sh
wget https://pastebin.com/raw/aYPfj3eB -O $scriptsdir/wifi-connect.sh
wget https://pastebin.com/raw/c0kJZeV2 -O $scriptsdir/lte-connect.sh

#Start/Stop By Hand
$MINIFI_HOME/bin/minifi.sh start
$MINIFI_HOME/bin/minifi.sh stop

#Install
$MINIFI_HOME/bin/minifi.sh install

In my vehicle this device gets a hard power down every time the vehicle’s ignition of turned off as it is powered from the accessory port. It has continued to operate and restart each time without issue in spite of this behavior.

Conclusion

While some functional needs for our use case are required to be written with shell-scripting (for now) there are a number of benefits related to working within MiNiFi.  Some may take it as a negative to have to use shell scripts to enable our functionality it should also been seen as the flexibility to leverage ALL of the functions that our System on Chip (Pi Zero-W) provides.  Out of the box your provided with first class back-pressure mechanics allowing edge devices to purge events that haven’t been able to send after they lose value or because the events are less valuable then others, and it includes both methods to prioritize events (FIFO/LIFO/etc), expire based on Event Count per queue, Storage Utilization, and Time. The Site-2-Site mechanism provides compression, encryption and delivery guarantees letting you focus on what matters and not getting the basics of Data Transfer correct. System Impact is almost non-existent for idle MiNiFi systems allowing it to fit on very small SOCs (Pi Zero-W 1CPU, 512MB Ram.)  The features just keeps getting better with some of what I had to do above already getting changed or improved!

Gotchas Notes Futures

  • Metrics are coming for your edge devices! 
  • Native GPSD MiNiFi Processors Incoming!
  • Soon there will be BiDirectional Communication at all parts – Edge/Cloud/DataCenter with the MiNiFi Command and Control (C2) Server
  • ARM still seems to be fresh even in 2017, be ready to compile libraries that are on x86 but not ARM.
  • If you looked at the LTE connection scripts you will see I have taken a very brute force approach to handling issues. Its likely there is significant room for improvement in this ENTIRE MiNiFi flow but for the point of demonstration its served its need.
    • That said, driving around I have had a few LTE failures that did not recover and need more investigation.
  • The Data Link used for Site-2-Site is decided on at this time by the OS. If we wanted to manipulate it may require modifying root, or extending S2S to target a specific interface. Currently we area using WiFi and LTE and grabbing whichever is available for data link at transfer time. There is significant room to make this process more ‘miserly’ (cost sensitive.)
  • Today there is no way to ‘identify’ that a specific set of data came from an edge device. It would be nice if all flowfiles were attributed with a specific host serial number that could be used downstream.
    • In NiFi, flowfiles received over S2S have s2s.hostname added as flowfile attributes.