Frequently in our IoT edge use-cases there is a need to transmit GPS data over varying types of data links such as cellular (3G/4G) and WiFi. This post will review how to setup MiNiFi CPP on a Raspberry Pi Zero-W with GPS, 4G Cellular Modem, and WiFi. Alot of the focus* here will be how you can use the MiNiFi product today to enable your SandBox platform which may result in this example feeling more ‘clunky’ to enable alot of actions the tracking use-case required this early on with MiNiFi.
*How to setup the Cloud/DataCenter NiFi servers for Site-2-Site is not discussed here.
WARN – The version of MiNiFi CPP used is 0.2; this is very early for this product and if your reading this Blog Post from a historical perspective it may be useful but its highly likely that if a year has past the product has matured more and would change most of the following designs and user experience for the better.
Edge & Platform Requirements
Many people ask me for my opinion on if the ‘Edge’ is part of the Spoke in a Hub & Spoke architecture. Given the increase in Edge computing I think of the edge as the Tire, or the Rubber on the road. I add this new layer into the traditional Hub & Spoke design because its an independent device that comes and goes based on jagged connectivity.
- Edge (TIRE) Requirements
- Captures DataPoints @ Given Sensor Intervals
- GPS, Cell Strength, etc
- Can be configured as N-Seconds or CRON like scheduling
- Back Pressure
- Can Prioritize what expires when storage becomes full
- Can Prioritize which messages in Queues/Connections to send downstream first
- Multiple Data Links (Cellular and WiFi)
- Links can be Encrypted (TLS)
- Delivery Guarantees
- Can Receive From Hubs (*In MiNiFi C2)
- Security
- The Hub and Edge both authenticate each-other with certificates
- Very Small Resource FootPrint of ‘Framework Software [MiNiFi]’
- See Prior Blog Post graphs at the Bottom for MiNiFi on Pi Zero-W with Idle System Metrics and even the workflow discussed here.
- Captures DataPoints @ Given Sensor Intervals
- Platform (HUB & SPOKE) Requirements
- Delivery Guarantees
- Back Pressure
- If a DataLink is down for days, keep the important data
- Transfer Throughput
- Transfers should perform AUTOMATIC Load Balancing, and discovery between nodes
- No Reverse Proxy
- No Load Balancer (Soft or Hard)
- Cluster AutoDiscover for Hosts to Load Balance on
- Compression
- Reduce egress costs
- Improve throughput (if spare cpu cycles exist)
- Transfers should perform AUTOMATIC Load Balancing, and discovery between nodes
- Security
- Edge, Spoke, and Hub all use certificates to authenticate
- Hub can PULL data from Spoke
- Prevents exposing the DataCenter (No Inbound Connections)
- Can use a HTTP_PROXY
- This makes many security folks happy to have a single audit point and is valid so long as the bandwidth is available
- Data Links can be Encrypted
Logical Architecture of the Platform
From an Architecture point of view each of these ‘zones’s is its own Architecture Failure Domain. This allows each of the Zones to fail without causing a outright failure to any of its connected systems. In the most simple of aspects the above Platform is utilizing a Hub & Spoke (and Tire) architecture where the DataCenter itself is the Hub, Cloud the Spoke… There are many reasons to go to this design; Security Surface reduction for multiple utility solutions needing data flow, and provides a bi-directional pipeline between the Cloud and DataCenter with Site-2-Site (See NiFi C2 for MiNiFi BiDirectional Comms), Regional Cloud for Latency, etc. Important to note that Site-2-Site capabilities in NiFi provides native load-balancing between clusters, compression, encryption, delivery guarantees for data, and if required can connect through an HTTP_PROXY to access clusters on other network segments.
Hardware
- Edge Device
- Pi Zero-W (using a U3 C10 Flash Storage Card)
- BU-353-S4 GPS Dongle
- Huawei Boltz 4G LTE Modem Unlocked
- Using Google Fi SIM
- Micro USB 2.0 Male to Female OTG Cable
- Anker USB Hub
- Cloud
- 4GB 2 Core Server
- DataCenter
- 8 Core 96GB Ram Server
Software
- Edge Device
- Raspbian GNU/Linux 8 (jessie)
- MiNiFi CPP 0.2 (built from source)
- GPSD
- usb-modeswitch
- libqmi
- Bash
- Cloud
- Ubuntu 16.04 Server
- NiFi 1.3
- DataCenter
- Ubuntu 16.04 Server
- NiFi 1.3
Code Flow and Implementation
The data-flow designed here is very basic with the goal of just getting GPS collected while in the field. Four Shell Scripts were built to provide the needed functionality and all directly link to the Site-2-Site Remote Process Group. All connections are configured as LIFO so that the most recent data is sent to the cloud whenever we regain connectivity and then the older data will get set when it can. Each connection was configured based on the important of the data so that GPS datapoints have up to 100MB of storage while simpler items like the WiFi connection status will only gather 10MB worth of events before expiring them. Additionally time expiration is being used; GPS will wait 168 hours before expiring events it has not sent, while other informational events expire every 48 hours.
Shell Scripts in ExecuteProcess Processors
If you have a desire to replicate the above flow you can find all there scripts and templates linked below. The scripts all generate JSON details about the actions they perform such as SCANNING for WiFi SSIDs, Resetting the entire 4G LTE Modem, Disk Metrics and even the GPS. The GPS JSON is generated by the GPSD application itself so we just capture it as is.
- Scripts
- Templates
Setting It all Up
Headless Pi Configuration
First image a new flashcard with the correct version of Raspbian.
dd if=raspbian-os-image.img of=/dev/disk2 bs=2m
To enable ssh by default drop a 0-byte file called ‘ssh’ into ‘/boot/’ this will enable ssh which on later versions of Raspbian is disabled by default for security.
touch /path/to/fashcard/boot/ssh
Configure to AutoJoin your WiFi SSID by editing /etc/wpa_supplicant_wpa_supplicant.conf, this is different the the WiFi script as this will occur at boot even if the scripts are not running. You will also have to configure your /etc/network/interface to autoconnect.
vi /etc/wpa_supplicant_wpa_supplicant.conf # network={ ssid="YourNetworkSSID" psk="Your Network's Passphrase" key_mgmt=WPA-PSK } # vi /etc/network/interfaces # You may want to use dhcp and not manual, your device may not be named wlan0 allow-hotplug wlan0 iface wlan0 inet manual wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf #
Install the GPS Daemon and have NTP sync time from GPS Satellites so time is kept even without a Data Link.
#Install Software sudo apt-get install gpsd gpsd-clients python-gps ntp #Setup GPSD Defaults #sudo vi /etc/default/gpsd # START_DAEMON="true" GPSD_OPTIONS="n" DEVICES="/dev/ttyUSB0" USBAUTO="false" GPSD_SOCET="/var/run/gpsd.sock" # #Test GPS sudo service gpsd restart cgps -s #Setup NTP to Sync from GPS Time sudo vi /etc/ntp.conf # # GPS Serial data reference server 127.127.28.0 minpoll 4 maxpoll 4 fudge 127.127.28.0 refid GPS fudge 127.127.28.0 time1 +0.140 # coarse processing delay offset # GPS PPS reference server 127.127.22.0 minpoll 4 maxpoll 4 prefer fudge 127.127.22.0 refid PPS fudge 127.127.22.0 flag3 1 # enable kernel PLL/FLL clock discipline # #Wait 5 Minutes and check if GPS is setting time sudo service gpsd restart sudo service ntp restart ntpq -p
Setup the 4G LTE Modem, this seems complex but isn’t to bad. If you Google you will find lots of other methods to do this including WVDail but here QMI will be used as it appears to be the currently most common method. Your device may hipup and require you to run modeswitch again. See the LTE-Connect script for a way to brute force handle problems.
#Install QMI, ppp, and usb_modeswitch apt-get install libqmi-utils ppp usb-modeswitch #Setup your Cell APN, below is configured for Google Fi. You will have to find the APN your cellular service uses. echo "APN=h2g2" >/etc/qmi-network.conf #Find the Huawei Modem Vendor and Product Info lsusb | grep Huawei Bus 001 Device 004: ID 12d1:1505 Huawei Technologies Co., Ltd. E398 LTE/UMTS/GSM Modem/Networkcard #If your using the same 4G modem as I am during the install its going to show up as 12d1:1505 but later it will become 12d1:1506 after using USB_SWITCH, this is expected. cd /etc/usb_modeswitch.d/ tar -xzvf /usr/share/usb_modeswitch/configPack.tar.gz 12d1\:1505 vi 12d1\:1505 # DefaultVendor= 0x12d1 DefaultProduct=0x1505 TargetVendor= 0x12d1 TargetProductList="140b,1506,150f" MessageContent="55534243123456780000000000000011062000000100000000000000000000" # sudo usb_modeswitch -c /etc/usb_modeswitch.d/12d1\:1505 sudo reboot #At each reboot use below to manually connect to WAN and renew your DHCPs sudo qmi-network /dev/cdc-wdm0 start sudo dhclient -r /dev/cdc-wdm0 sudo dhclient/dev/cdc-wdm0 sudo qmi-network /dev/cdc-wdm0 status #wait a few minutes and check for an IP, then test the interface ifconfig ping -I wwan0 www.google.com #you can also use traceroute -i $interface www.google.com
It Lives!
Depending on the chipset where MiNiFi CPP is installed you may be required to compile it from source code (as was the case for the Raspberry Pi as its ARM.) If your testing this on an x86 system you can just download the tarball. The tarball folder absolute path becomes $MINIFI_HOME. Because the scripts in the ExecuteProcess Processors take care of starting most everything the services need such as GPS Daemons init.d scripts are not required for everything OTHER THAN the MiNiFi service itself; you can install MiNiFi for auto-start: $MINIFI_HOME/bin/minifi.sh install
https://nifi.apache.org/minifi/download.html
#RHEL/CentOS: yum install -y epel-release yum install -y leveldb #Debian/Ubuntu: apt install -y libleveldb-dev apt install -y libxml2 #Reuse Existing Template and Make sure to update the Site-2-Site Host and Script locations $MINIFI_HOME/conf/config.yml wget https://pastebin.com/raw/E52LvF2b -O $MINIFI_HOME/conf/config.yml vi $MINIFI_HOME/conf/config.yml # Update Host and Port of Site-2-Site Remote Processing Groups: - name: http://r01s01n01:8080/nifi url: http://r01s01n01:8080/nifi comment: '' timeout: 30 sec yield period: 10 sec Input Ports: - id: 089c253b-015d-1000-2077-c6fa9242c2b9 name: rx comment: '' max concurrent tasks: 1 use compression: false Properties: Port: 12551 Host Name: r01s01n01 # #By default all the scripts are located at '/iot/scripts/' export scriptsdir=/iot/scripts mkdir -p $scriptsdir wget https://pastebin.com/raw/RcdyQGTJ -O $scriptsdir/disk-report.sh wget https://pastebin.com/raw/n4NiXnE9 -O $scriptsdir/gps-logger.sh wget https://pastebin.com/raw/aYPfj3eB -O $scriptsdir/wifi-connect.sh wget https://pastebin.com/raw/c0kJZeV2 -O $scriptsdir/lte-connect.sh #Start/Stop By Hand $MINIFI_HOME/bin/minifi.sh start $MINIFI_HOME/bin/minifi.sh stop #Install $MINIFI_HOME/bin/minifi.sh install
In my vehicle this device gets a hard power down every time the vehicle’s ignition of turned off as it is powered from the accessory port. It has continued to operate and restart each time without issue in spite of this behavior.
Conclusion
While some functional needs for our use case are required to be written with shell-scripting (for now) there are a number of benefits related to working within MiNiFi. Some may take it as a negative to have to use shell scripts to enable our functionality it should also been seen as the flexibility to leverage ALL of the functions that our System on Chip (Pi Zero-W) provides. Out of the box your provided with first class back-pressure mechanics allowing edge devices to purge events that haven’t been able to send after they lose value or because the events are less valuable then others, and it includes both methods to prioritize events (FIFO/LIFO/etc), expire based on Event Count per queue, Storage Utilization, and Time. The Site-2-Site mechanism provides compression, encryption and delivery guarantees letting you focus on what matters and not getting the basics of Data Transfer correct. System Impact is almost non-existent for idle MiNiFi systems allowing it to fit on very small SOCs (Pi Zero-W 1CPU, 512MB Ram.) The features just keeps getting better with some of what I had to do above already getting changed or improved!
Gotchas Notes Futures
- Metrics are coming for your edge devices!
- Native GPSD MiNiFi Processors Incoming!
- Soon there will be BiDirectional Communication at all parts – Edge/Cloud/DataCenter with the MiNiFi Command and Control (C2) Server
- ARM still seems to be fresh even in 2017, be ready to compile libraries that are on x86 but not ARM.
- If you looked at the LTE connection scripts you will see I have taken a very brute force approach to handling issues. Its likely there is significant room for improvement in this ENTIRE MiNiFi flow but for the point of demonstration its served its need.
- That said, driving around I have had a few LTE failures that did not recover and need more investigation.
- The Data Link used for Site-2-Site is decided on at this time by the OS. If we wanted to manipulate it may require modifying root, or extending S2S to target a specific interface. Currently we area using WiFi and LTE and grabbing whichever is available for data link at transfer time. There is significant room to make this process more ‘miserly’ (cost sensitive.)
- Today there is no way to ‘identify’ that a specific set of data came from an edge device. It would be nice if all flowfiles were attributed with a specific host serial number that could be used downstream.
- In NiFi, flowfiles received over S2S have s2s.hostname added as flowfile attributes.