WWAN (LTE) on HPE EL300 with Linux and Sierra Wireless EM7565
This is the second of two posts discussing the use of wireless technologies with HPE’s Edgeline EL300 Converged Edge System. The last post was less EL300-specific and more about enabling wireless in Ubuntu server, while this post is more specific to the EL300 yet general enough to apply to many WWAN-enabled systems.
- WiFi on HPE EL300 with Ubuntu 18.04
- WWAN (LTE) on HPE EL300 with Ubuntu 18.04 and Sierra Wireless EM7565
Late last year, HPE came to market with new technology for the digital transformation of Operational Technology (OT) environments. Part of this release includes a new alternative form factor for an edge compute node, namely the EL300. This device is a ruggedized, fanless system for adverse condition environments, and is capable of employing a number of unique features with respect to traditional systems like:
- 4G/5G Connectivity
- Bluetooth Connectivity
- Modules for:
- Time-Sensitive Networking (TSN)
- General Purpose I/O (GPIO)
There are a couple of links to EL300 information in the references section at the bottom of the page for more information.
We employed the EL300 for use in one of our customer’s OT environments to do video inferencing at the edge. The software stack that I needed to run on this box was best supported by running Ubuntu 18.04 Server, which HPE doesn’t officially support, so this activity was a bit of off-roading. But being Ubuntu, this post should apply to many systems.
When starting this project I thought to enable WWAN was going to be an interesting, relatively straight forward activity. As it turns out, it WAS NOT straight forward, so I’m hoping this blog will help someone going through a similar journey. As discussed in the last blog, Ubuntu 18.04 utilizes netplan for network configuration, which doesn’t do well with wireless technologies, for which NetworkManager is the better solution. This WWAN config will be using NetworkManager, so if you didn’t go through the last blog, jump back to that and go through the installation of NetworkManager, and then come back here.
So from here, it’s expected that NetworkManager is installed. In addition, you’ll need the following:
- Sierra Wireless EM7565 card
- SIM card from your wireless provider
The first thing to do is to install the wireless modem and the SIM card. Here’s where the wireless card goes when you open up the underside of the unit, and where the SIMs are located and how I found them to be enumerated. The SIM slots are spring-loaded, so you’ll be able to feel when it catches.
In working through this configuration I came across A LOT of conflicting information, so let me share some thoughts about THIS SPECIFIC CONFIG:
- There’s no need to use AT commands.
- WWAN cards are modems. NetworkManager uses ModemManager to manage these cards. When you install NetworkManager, ModemManager is also installed as a dependency.
- Vendor-specific firmware is required. In my case, my provider is Verizon, so I installed the Verizon-specific firmware.
- I tried installing the firmware from Ubuntu using a few different tools but wasn’t able to get it to work. I found some threads suggesting that Windows was the best platform for that, so I will be using that method.
So as to hopefully make this process as clear as possible, let’s deal with the Windows aspect before configuring anything in Ubuntu. What I found easiest for this, rather than going through a Windows install, only to replace it immediately afterward, is to create a USB drive that contains Windows To Go, a portable version of Windows that can be booted from a USB drive. In this regard, I won’t be detailing the procedure for creating a Windows To Go stick since there are a number of resources already available. Here are some tutorials:
Also, on a bit of a tangent, when doing these types of activities with USB sticks, it’s really not fun to sit there and wait for the slow I/O performance of some of the sticks. If you want to get better performance and complete these tasks faster, check out https://usb.userbenchmark.com/. They’ve got a USB section that has benchmarks for many of the USB sticks available. Find one that’s good and fast and use the model number to purchase from your favorite retailer.
Now, having created your handy, dandy Windows To Go USB stick, boot the EL300 from it. Once in Windows, you’re going to need to download the firmware and Windows drivers. Make sure you download the firmware that’s specific to your carrier; in my case, Verizon. Unfortunately, when I did this process I didn’t take screenshots, and I won’t have access to the machine for another week, so I’ll convey the steps and update them with more detail at a later date.
- Install Windows drivers for the Sierra Wireless EM7565 card
- Install the vendor-specific firmware for the Sierra Wireless EM7565 card
- Go to Cellular Settings and select the SIM. I used SIM 1 which is under the WiFi card.
- Validate that the card connects to the carrier (supported OS is easier to troubleshoot)
With that process, I had no issue connecting to the Verizon network from within Windows To Go. Once validated, shut down the machine, remove the USB stick and boot back into Ubuntu.
As mentioned in the previous blog, with Ubuntu 18.04 we let netplan manage the wired interfaces and have NetworkManager manage the wireless ones. NetworkManager uses the file
/usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf to manage control of interfaces. By default it looks like this:
This particular card doesn’t present itself as type wwan, but rather gsm. So to give NetworkManager control over it we need to tweak the file:
If you miss this step when you try to configure you will receive an error from NetworkManager that “Activation failed because the device is unmanaged”.
The EM7565 requires more updated QMI software than is included with Ubuntu 18.04 (per https://github.com/danielewood/sierra-wireless-modems#basic-setup). Let’s use the latest versions instead of his recomendations (go to libqmi in the Universe repository and libqmi in the Main repository to determine the proper version numbers):
# wget http://security.ubuntu.com/ubuntu/pool/universe/libq/libqmi/libqmi-utils_1.22.0-1.2_amd64.deb # wget http://security.ubuntu.com/ubuntu/pool/main/libq/libqmi/libqmi-glib5_1.22.0-1.2_amd64.deb # wget http://security.ubuntu.com/ubuntu/pool/main/libq/libqmi/libqmi-proxy_1.22.0-1.2_amd64.deb # sudo dpkg -i libqmi-* # sudo apt install -f # dpkg -l | grep qmi
Now let’s restart NetworkManager and see what we can find from the modem (note that ModemManager uses mmcli, while NetworkManager uses nmcli):
# sudo systemctl restart NetworkManager # sudo mmcli --modem 0 /org/freedesktop/ModemManager1/Modem/0 (device id '10de51cedef75f0211e4b4e3c09ca8e0701a3cc1') ------------------------- Hardware | manufacturer: 'Sierra' | model: 'MBIM [1199:90B1]' | revision: 'SWI9X50C_01.08.04.00' | supported: 'gsm-umts, lte' | current: 'gsm-umts, lte' | equipment id: '393576110206380' ------------------------- System | device: '/sys/devices/pci0000:00/0000:00:14.0/usb2/2-3' | drivers: 'cdc_mbim' | plugin: 'Sierra' | primary port: 'cdc-wdm1' | ports: 'wwp0s20f0u3 (net), cdc-wdm1 (mbim)' ------------------------- Numbers | own : '+1##########' ------------------------- Status | lock: 'none' | unlock retries: 'sim-pin2 (3)' | state: 'registered' | power state: 'on' | access tech: 'lte' | signal quality: '35' (recent) ------------------------- Modes | supported: 'allowed: 3g, 4g; preferred: none' | current: 'allowed: 3g, 4g; preferred: none' ------------------------- Bands | supported: 'unknown' | current: 'unknown' ------------------------- IP | supported: 'ipv4, ipv6, ipv4v6' ------------------------- 3GPP | imei: '393576110206380' | enabled locks: 'fixed-dialing' | operator id: '311480' | operator name: 'Verizon Wireless' | subscription: 'unknown' | registration: 'home' ------------------------- SIM | path: '/org/freedesktop/ModemManager1/SIM/0' ------------------------- Bearers | paths: '/org/freedesktop/ModemManager1/Bearer/0'
Now, that all looks good. We can see the phone number assigned to the SIM, the IMEI number of the device, etc. Lots of good info here, all from the modem. Now let’s create a connection and try to connect. Before we do that, you’re going to need to identify the APN (Access Point Name) that your carrier uses. I found the Verizon info here.
# sudo nmcli con add con-name wwan-verizon ifname cdc-wdm1 type gsm apn vzwinternet ipv4.method auto connection.autoconnect yes # sudo cat /etc/NetworkManager/system-connections/wwan-verizon ... # sudo nmcli radio wwan on # sudo nmcli device ... # sudo nmcli con up wwan-verizon # sudo nmcli con NAME UUID TYPE DEVICE wifi-jafar d9ec11c9-ec43-4584-90e1-bbdeb5a0af36 wifi wlp6s0 wwan-verizon 129f60d9-341e-4cdf-8d63-162a869652ab gsm cdc-wdm1 # sudo mmcli --modem 0 /org/freedesktop/ModemManager1/Modem/0 (device id '10de51cedef75f0211e4b4e3c09ca8e0701a3cc1') ... ------------------------- Status | lock: 'none' | unlock retries: 'sim-pin2 (3)' | state: 'connected' | power state: 'on' | access tech: 'lte' | signal quality: '38' (cached) ------------------------- ... # sudo mmcli --bearer 0 Bearer '/org/freedesktop/ModemManager1/Bearer/0' ------------------------- Status | connected: 'yes' | suspended: 'no' | interface: 'wwp0s20f0u3' | IP timeout: '20' ------------------------- Properties | apn: 'vzwinternet' | roaming: 'allowed' | IP type: 'ipv4v6' | user: 'none' | password: 'none' | number: '*99#' | Rm protocol: 'unknown' ------------------------- IPv4 configuration | method: 'static' | address: '100.103.222.7' | prefix: '28' | gateway: '100.103.222.8' | DNS: '220.127.116.11', '18.104.22.168' | MTU: '1428' ------------------------- IPv6 configuration | method: 'static' | address: '2600:1012:b023:896:d0e1:f349:45c7:5024' | prefix: '64' | gateway: '2600:1012:b023:896:7d60:9391:dfab:e549' | DNS: '2001:4888:64:ff00:620:d::', '2001:4888:62:ff00:62e:d::' | MTU: '1428' ------------------------- Stats | Duration: '120' | Bytes received: 'N/A' | Bytes transmitted: '552'
Terrific! We’re connected, we have IP addresses, and can talk out! We’re good to go — well, sort-of!
My intention in setting this up was so that I could remotely manage the EL300 via the cellular broadband network. It turns out (and your mileage can vary between carriers and cellular plans) that I need a more evolved configuration in order to accomplish this. You see:
- The network that my card is connecting to at present is 100.103.222.7/29
- That address falls squarely into the CIDR 100.64.0.0/10 for which WHOIS reveals that it’s a shared address space for ISPs:
- Since this is a headless Linux system, issuing a
curl --interface wwp0s20f0u3 ifconfig.me(the wwp0s20f0u3 interface is my Verizon WWAN connection) reveals that “my address” from the outside resolves to 22.214.171.124 (ATM) which suggests that I’m being NATted and am not directly accessible.
- And a little more digging uncovered this: Carrier-grade NAT
So, while I am properly connected, I don’t have inbound access. One way to make this work would be to start a VPN client upon connection to the cellular network. But that’s too much to cover in this blog. Maybe another time…
But, something else that should be covered is routing. You see, when you have multiple network connections, you need to take routing into account. If a connection were to come in over the WWAN interface, it may cause problems if the response goes back out over the Ethernet connection (typical, default config). To deal with this, we need to use a feature of iproute2 that allows for multiple routing tables. The following sites are good resources on the topic:
Unfortunately, I didn’t have time to take this to fruition. The real way to do this would be to have a script run when a WWAN connection is established and dynamically change the info in a routing table specific to that connection. Here are the pieces required to accomplish the task:
- Create a routing table for the WWAN device:
# sudo vi /etc/iproute2/rt_tables.d/verizon.conf 200 verizon
- When the link is established:
# sudo ip rule add from 100.103.175.116 dev wwp0s20f0u3 table verizon # sudo ip route add default via 100.103.175.117 dev wwp0s20f0u3 table verizon
- When the link is taken down:
# sudo ip route del default via 100.103.175.117 dev wwp0s20f0u3 table verizon # sudo ip rule del from 100.103.175.116 dev wwp0s20f0u3 table verizon
- And here are some commands to help troubleshoot and lot when changes occur:
# sudo ip route show table verizon # sudo ip route show table main # sudo ip route show table local # sudo ip route flush table verizon
So that’s the main content I’ve got. Below is some additional information I gathered along the way that might be of some use to someone. If you have any questions or feedback, I’d love to hear from you. Just jump into Discourse via the link below and let me know your thoughts.
Until next time, cheers!
https://techship.com/faq/how-to-step-by-step-set-up-a-data-connection-over-qmi-interface-using-qmicli-and-in-kernel-driver-qmi-wwan-in-linux/ has a lot of good information, but it’s not accurate to our current setup. It is not necessary to load the qmi_wwan driver. Instead, you can target the cdc_mbim device. The following commands are ones that I’ve had success with — the
--umi-get-card-status yields good info since there are two SIM slots in the EL300.
# sudo qmicli --device=/dev/cdc-wdm1 --device-open-proxy --get-wwan-iface # sudo qmicli --device=/dev/cdc-wdm1 --device-open-proxy --wda-get-data-format # sudo qmicli --device=/dev/cdc-wdm1 --device-open-proxy --dms-get-manufacturer # sudo qmicli --device=/dev/cdc-wdm1 --device-open-proxy --dms-get-model # sudo qmicli --device=/dev/cdc-wdm1 --device-open-proxy --dms-get-revision # sudo qmicli --device=/dev/cdc-wdm1 --device-open-proxy --dms-get-ids # sudo qmicli --device=/dev/cdc-wdm1 --device-open-proxy --uim-get-card-status
Here are some ModemManager commands that might be helpful:
# sudo mmcli --list-modems # sudo mmcli --modem 0 # sudo mmcli --modem 0 --list-bearers # sudo mmcli --bearer 0 # sudo mmcli --sim (found with "mmcli --modem 0" or "qmicli --device=/dev/cdc-wdm1 --device-open-proxy --uim-get-card-status" (zero-based))
The kernel modules that are in play are:
# lsmod | grep cdc cdc_mbim 16384 0 cdc_wdm 20480 2 cdc_mbim cdc_ncm 36864 1 cdc_mbim usbnet 45056 2 cdc_mbim,cdc_ncm cdc_acm 32768 0
The EL300 includes an integrated System Manager (iSM) that can be used to manage the system outside of the main OS; similar to HPE iLO. The wireless card can be used for either the main OS or the iSM, but not both at the same time. Unloading the cdc_mbim module (
modprobe -r cdc_mbim) releases the device so the iSM can use it for a management connection (without reboot).
- HPE Edgeline Converged Edge System Platform Launch
- HPE Edgeline EL300 with OT Link…
- nmcli man page
- mmcli man page
- Launchpad Bug Entry that gave me some insight into the process above
- Ubuntu – configure-cellular-connections
- EM7565 Product Page
- EM7565 Direct Firmware Link
- Verizon KB-46578/
- Carrier-grade NAT