You ever feel like you just want to make some noise? Maybe a horrendous noise? And you want to use electronics to do it? Well do I have some fun projects for you, lets dive into the world of simple analog noise makers!

Overview

Nowdays most intro-to-sound tutorials will have you grab an arduino or other microcontroller and a speaker, but back in the day, things were still analog. No computer, just a bunch of electrical components to use (or abuse) to create oscillating voltages with different frequencies to make sound. This eventually branched out into synthesizers and electronic music we’ve all come to know and love. But you don’t have to drop $450 on an analog synthesizer in order to make some loud, synth-like sounds.

The most basic - Super Simple Oscillator

Probably the simplest I’ve found is this super simple oscillator from Look Mum No Computer. This is created by abusing a poor transistor that doesn’t know whats coming for em.

How it works

The basic concept has you hook up a transistor’s collector and emittor backwards in the circuit, in parallel with a capacitor, with a resistor in series. And just to be even meaner to the transistor, you cut off the base middle pin (this helps prevent it from interfering with your shenanigans). The battery charges up the capacitor while the transistor basically has nothing flowing through it because it’s backwards. Once the capacitor reaches the right level of voltage, though, it triggers an “avalanche” in the transistor, where it suddenly has very low resistance. The capacitor discharges through the transistor until its voltage is lower than that trigger point, and the circuit goes back to its original state, with the capacitor charging up. This repeats, and you get some oscillation, and therefore, noise! If instead of a single resistor, you do a resistor plus potentiometer, you get the ability to control how fast the capacitor charges, and therefore the frequency of the oscillation.

The fun part of this circuit is:

• if you have any sort of component collection, you probably already own all the parts for it (I did!)
• exactly which value of each component is super loose. You can use different NPN transistors, which will trigger at different voltages. You can use different sized capacitors, which determine the baseline frequency your pot is changing. You can add an LED, or not.

After playing around with it, it kind of felt like you could just throw a handful of random components on it, and it’d somehow still oscillate. Fun stuff! I messed around with this circuit a lot on my breadboard, and did some double oscillator for two tones sort of thing as well:

However, there’s definite limitations to what sort of sounds you can get out of it. So let’s go a little bigger.

Next Step Up - Atari Punk Console

While it’s not individual components, a classic 555 timer chip counts as analog to me. So how about two of them? Lets build the Atari Punk Console.

NOTE There’s a lot of different circuits and walk throughs online. If you want to see how variable they can be, it’s fun to compare the circuits with different values and compare/contrast. Well, fun to me, anyway.

How it works

I ended up using the circuit from this website: https://www.build-electronic-circuits.com/atari-punk-console/

Mainly because it included some resistors that other versions lack. For example without, look at the circuit this user built on electronics stack exchange. This version of the Atari Punk Console has the second potentiometer conect the power source to pin 7 on the second 555 with no resistor in series. So if the potentiometer is turned all the way down, the resistance is 0 and the pin is getting full power. So when the 555’s internal transistor discharges through the chip, it’s just shorting out. Schematic shown below.

Fun Fact Creating shorts across your battery is not good!

The linked stack exchange talks about what happens then, the timer gets very hot or even melts. Not so good for music making.

Throwing in a few resistors so there’s always some resistance on the connections between power and the potentiometer seemed like a good thing, so I went with that version. I also wanted a way to turn it off when I was done, though, so I added a switch and LED in-line with the power. The base schematic I ended up using:

My Build

After spending a concerningly long time at our local surplus store with a huge electronic component selection (Ax-man surplus, how I love you), I had everything I needed.

I translated the schematic to a breadboard for initial testing, and it came together suprisingly easily.

Once I had proof of concept, I converted it over to a perfboard with soldered connections for a more permanent setup. Below are my in-progress photos as I went.

Getting started:

Partially finished with the perfboard layout, and a lot of messy long wires:

All the components soldered together and working! Now I just need an enclosure…

I had a hackerbox subscription a while back, and their kits come in fun skull-and-circuit decorated boxes. Which seemed perfect for something called a punk console:

And just like that, the first version of the Atari Punk Console was done! But, well, we can do better, now can’t we…

Modifications

Switching Headphone Jack

My partner occasionally works from home. It turns out listening to loud annoying sounds repeatedly gets annoying. So I modified the original schematic to include a switching audio jack, courtesy of ax-man surplus. The jack is a standard headphone jack, and it has 3 connection points. When there’s no headphones plugged in, electricity will go through one path, and when headphones are plugged in, it goes through a different path. So if it’s spliced into the path to the speaker, I can set it up so when I plug in headphones, it goes through my headphones, and when I unplug them, it goes back out through the speakers.

I can now listen to my atari punk console quietly! Schematic change:

CV Control

The 555 chip has a CV pin right on it, pin 5, which isn’t used at all for most atari punk consoles I’ve seen. However, it is possible to add that functionality pretty easily. This allows you to add some audio jack plugs that, when connected to an appropriate control voltage source (e.g. certain sythensizers), can control the exact tone you get. The addition is shown below.

As you can see, it’s adding a vactrol connected to an audio jack on one side, and the CV voltage pin on the other. The vactrol is just a photoresistor and an LED, so as the voltage goes up, the LED gets brighter, and the photoresistor changes resistance. The end result becomes: “Voltage change from the CV input results in a voltage change on the CV pin”. But with bonus that it keeps the atari punk console isolated from whatever is plugged into the CV input jack, since only light is being sent between the CV jack and the atari punk console.

Now, I didn’t actually buy a fancy vactrol chip or anything, and instead did the DIY version. This can be made by using a photoresistor and a LED, with some heat shrink to hold them together and keep out any light. And tada, vactrol!

NOTE If you learn better via video, you can see Look Mum No Computer make one here: https://youtu.be/Ch9w5JtbZSc?si=L04PQ_3qpAdYQGnf&t=215

Final Results

With some small mods to the enclosure, my new and improved atari punk console was complete! The two CV input jacks I put on the left hand side, with the headphone out jack in the lower right with a little label, so I didn’t forget which was which. I also added some cuts with an xacto knife over where the speaker was mounted in the upper right, to stop the cardboard from muffling the sound. And since I just free-handed it, that certainly gives it a, shall we say, punk aesthetic.

And if you want to hear what noises it can play, here’s a short recording. I’m simply turning the potentiometers in various ways to get the different tones.

Next Steps

I started to play with the CV input, enough to prove it can control the 555 timers. However, I got distracted and decided to try building a 8-step sequencer to work with the atari punk console, and then started debating about adding a gate control so the sound on/off can be controlled by a separate syth of some type… So maybe that will be some future blog posts! In the meantime, I’ll just occasionally pull this out when I feel the need to make some good buzzy analog sounds.

Resources

TI’s 555 timer documentation: https://www.ti.com/lit/ds/symlink/lm555.pdf

Atari Punk Console Schematic https://www.build-electronic-circuits.com/atari-punk-console/

Look Mum No Computer Super Simple Oscillator: https://www.lookmumnocomputer.com/simplest-oscillator/

Look Mum No Computer DIY Vactrol explanation: https://youtu.be/Ch9w5JtbZSc?si=L04PQ_3qpAdYQGnf&t=215

Since I already talked about the Remote ID protocol and how it was broadcast, you might now want to actually listen to those Remote ID messages around your home. There’s certainly commercial-grade solutions for that, but maybe you don’t want to drop a thousand dollars or book a meeting with a company so you can buy it, all for a fun sidequest. Instead, lets look for some slightly cheaper and easier to access options, and see what Remote ID functionality they support, and what they don’t.

First up: The first drone detection hardware/firmware I came across, Colonel Panic’s Sky-Spy! I’ll review the hardware and firmware functionality, how to DIY the hardware if you’re feeling cheap, and include a bonus of some forked code that I’ve started working on!

…I had originally planned on doing one overview post of all current available options…but then I got nerdsniped. So I’ll be going more in-depth on each.

NOTE Highly recommend reading my previous post about the remote ID transmission types, if you haven’t already. This post assumes you know about that already.

Overview - Functionality and Code Review

Colonel Panic has several fun pieces of hardware, including their OUI-SPY, which was originally setup as a general BLE scanner, but was re-purposed by them as a drone detector as well. The custom hardware currently uses a Xiao ESP32-S3, a piezo buzzer, and custom PCBs to make a nice little handheld device. They used their hardware with a custom code base called Sky-Spy (The image above is the logo from that firmware). Colonel Panic has a lot of other cool firmware setup for the OUI-Spy as well, such as the Unified Blue firmware that combines a lot of different functionality into one codebase.

In the previous post, I talked about all the potential ways drones can broadcast: BLE 4 and 5 advertising packets, and 2.4/5G Wifi beacon and NAN frames. Based on my research, the OUI-Spy covers 3 out of possible 6 transmission types. How do I know? By looking at the current hardware and firmware. Lets take a look!

NOTE They’re pretty active with development, so some of my research may be moot soon. BUT if you know how and where to look for this info now, you’ll be able to take a look at the latest code and compare!

BLE 4 and 5

For BLE, we first check if the hardware can handle 4 and 5, then look at the code. The hardware is easy, espressif specifically says ESP32S3’s can use both BLE 4 and 5.

For the firmware side, we can see it’s using pretty standard espressif setup for BLE advertisements. However, it’s not setup for the BLE 5 extended advertising packets, which can be checked by looking at the BLE 5 example code from espressif.

Want to look at the code and follow along? The main Sky-spy code can be found here: https://github.com/colonelpanichacks/Sky-Spy

Here’s the callback in Sky-Spy:

class MyAdvertisedDeviceCallbacks : public BLEAdvertisedDeviceCallbacks {
public:
  void onResult(BLEAdvertisedDevice device) override {
    int len = device.getPayloadLength();
    uint8_t* payload = device.getPayload();
    // payload parsing removed here for brevity...
  }
};

And the BLE initialization in Sky-Spy:

  BLEDevice::init("DroneID");
  pBLEScan = BLEDevice::getScan();
  pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks());
  pBLEScan->setActiveScan(true);

Lets compare that with the Espressif example code for extended advertising.

NOTE The example code lives here: https://github.com/espressif/arduino-esp32/blob/master/libraries/BLE/examples/BLE5_extended_scan/BLE5_extended_scan.ino

Here’s Espressif’s callback, note it’s inheriting from a differenc class, BLEExtAdvertisingCallbacks, and the onResult argument type is different:

class MyBLEExtAdvertisingCallbacks : public BLEExtAdvertisingCallbacks {
  void onResult(esp_ble_gap_ext_adv_report_t report) {
    if (report.event_type & ESP_BLE_GAP_SET_EXT_ADV_PROP_LEGACY) {
      // here we can receive regular advertising data from BLE4.x devices
      Serial.println("BLE4.2");
    } else {
      // here we will get extended advertising data that are advertised over data channel by BLE5 devices
      Serial.printf("Ext advertise: data_le: %u, data_status: %u \n", report.adv_data_len, report.data_status);
    }
  }
};

And here’s the initialization for extended advertising, note it’s setting the extended scan functions:

 BLEDevice::init("");
  pBLEScan = BLEDevice::getScan();  //create new scan
  pBLEScan->setExtendedScanCallback(new MyBLEExtAdvertisingCallbacks());
  pBLEScan->setExtScanParams();         // use with pre-defined/default values, overloaded function allows to pass parameters
  delay(1000);                          // it is just for simplicity this example, to let ble stack to set extended scan params
  pBLEScan->startExtScan(scanTime, 3);  // scan duration in n * 10ms, period - repeat after n seconds (period >= duration)

So this research shows: Sky-Spy is capturing the BLE standard advertising packets, but not the extended ones.

However, the espressif example appears to get both legacy and extended advertising, which means this hardware could actually capture both. I plan on looking into this for my custom fork of this code (more below).

5G Wifi

For determining if it supports 5G, we once again start with looking at the hardware. The ESP32-S3 hardware fact sheet shows it does not support 5G wifi. So this research shows: The ESP32-S3 only has 2.4 GHz Wi-Fi and BLE 5.

The new ESP32-C5 does support 5GHz wifi, and according to Espressif is the “first RISC-V MCU that supports 2.4 and 5 GHz dual-band Wi-Fi 6, along with Bluetooth 5 (LE) and IEEE 802.15.4 (Zigbee, Thread) connectivity”. So this may be a hardware upgrade for the future. And in fact Colonel panic’s repo has a sub-folder that looks to include some work done on the XIAO seeed ESP32C5 https://github.com/colonelpanichacks/Sky-Spy/tree/main/xiao-c5-5g. So maybe this is a future upgrade for OUI-Spy?

2.4G Wifi

We’ve already confirmed the ESP32S3 has 2.4 GHz Wi-Fi, so lets jump right to the firmware. Here we have to confirm both NAN and beacon frames are parsed.

First, the initialization, setting up the wifi module to be in promiscuous mode and scanning Channel 6:

  WiFi.mode(WIFI_STA);
  WiFi.disconnect();
  
  esp_wifi_set_promiscuous(true);
  esp_wifi_set_promiscuous_rx_cb(&callback);
  esp_wifi_set_channel(6, WIFI_SECOND_CHAN_NONE);

Looking at the callback function it references, it checks the packets to see if they’re NAN or beacon and parses them as needed (snippet below). The first if statement short circuits if the wifi packet isn’t a management packet. It then checks if the destination address is the expected NAN address, or if the management frame’s type is 0x80, the beacon type.

void callback(void *buffer, wifi_promiscuous_pkt_type_t type) {
  if (type != WIFI_PKT_MGMT) return;
  
  wifi_promiscuous_pkt_t *packet = (wifi_promiscuous_pkt_t *)buffer;
  uint8_t *payload = packet->payload;
  int length = packet->rx_ctrl.sig_len;
  
  static const uint8_t nan_dest[6] = {0x51, 0x6f, 0x9a, 0x01, 0x00, 0x00};
  if (memcmp(nan_dest, &payload[4], 6) == 0) {
    if (odid_wifi_receive_message_pack_nan_action_frame(&UAS_data, nullptr, payload, length) == 0) {
      // parsing removed for brevity...
    }
  }
  else if (payload[0] == 0x80) {
   // parsing removed for brevity...
  }
}

Using those two things as the way to check for NAN or Beacon frames is a bit odd, but does show Sky-Spy parses Wifi NAN and Beacon Frames in 2.4G Wifi.

Parsing Remote ID protocols

Something I like about this codebase is that it uses the open remote ID library: https://github.com/opendroneid/opendroneid-core-c for parsing the received messages. This parsing library was originally put together by Gabriel Cox, who was “leading the ASTM UAS Remote ID workgroup (as Chairman) consisting of industry and government stakeholders to bring a consensus standard solution and Means of Compliance (MOC) for Remote ID.” (quote pulled from his company website). It also claims in the README to be compliant with the updated F3411-22a, the version of the specification that currently does not appear to be available online unless you pay ASTM for the pleasure.

I wanted to call this out specifically, because some other solutions look to use custom parsing instead of a library, which has its own trade-offs.

Sky-Spy Functionality Conclusion

You can see above then that the Sky-Spy firmware and the OUI-Spy can detect a subset of ways the drones transmit remote IDs: BLE 4, and 2.4G Wifi Beacon and NAN frames. This means, for example, DJI drones that broadcast on 2.4G Wifi will be heard, but Skydio drones that use 5G-only will not. With the custom hardware currently $85 on Colonel Panic’s website, that may be too steep a price for you, for the functionality given. If you’re feeling handy (which you may be, since you’re reading this), I do want to mention Colonel Panic does give enough info you can do a DIY OUI-Spy device, which may make the trade-off worth it for you.

Setup with DIY Hardware

When I was first interested in this, the official OUI-Spy Hardware was estimated a month or so out. However, Digikey sells the parts that Colonel Panic mentions as a DIY option, and has a much faster turnaround. So with a little bit of perf board and soldering, I had a functionally-identical bit of hardware that cost roughly 10 dollars and arrived faster. Downside, of course, is that I had to do a lil bit of soldering and firmware flashing. Luckily, that’s my jam.

I picked up a 3-pack of the ESP32S3’s, since it was a bit cheaper per-board: https://www.digikey.com/en/products/detail/seeed-technology-co-ltd/102010573/24814471

And piezo buzzers are pretty interchangeable, but I grabbed these: https://www.digikey.com/en/products/detail/tdk/PS1720P02/935932

Then you just connect the buzzer to the ESP32-S3 on the right pins (ground and the pin the firmware uses). To keep it all together, I used a small chunk of bakelite perf board I had bought ages ago from adafruit, and some solid core wire. I routed the wire under the ESP32-S3, so it was as small of footprint as I could make it.

With the hardware put together, I just had to flash the firmware onto it. I had to make sure VSCode was installed, and the platform IO extension was loaded in VSCode. Then I cloned the Sky-Spy repo, and did a basic platform IO build and upload, while my device was plugged in with a data USB-C cable. And voila, a functioning drone detector!

Below are some pictures of my final product!

Note I’ve added a larger antenna as my external antenna for the Wifi, since I had one lying around. You can just use the small antenna that comes with the Xiao ESP32-S3, you’ll probably just get a smaller range with it.

And the back side to see my “better than good, it’s good enough” soldering on the bit of perf board I used:

My Modified Codebase

I can never leave well enough alone, so I forked Colonel Panic’s codebase and started adding support for other hardware types. Codebase is here. So far I’ve added and tested the Adafruit TFT reverse feather and a Lilygo T Dongle S3.

I also plan to upgrade the codebase to support both legacy and extended advertising BLE packets. As well as take some of the code changes from the Flock-you codebase that Colonel Panic also made. Mainly writing to memory for detection retention through power-loss.

Firmware Change 1 - Add support for a display

I’ve updated the code to use a DisplayHandler interface that gets implemented by the specific bits of hardware. This then lets the main.cpp pull in the correct implementation at the top, and then just use the interface functions to interact with it in the code. This cuts down on the number of #ifdefs in the code and makes it easy to add more hardware going forward. So for example, my Adafruit board creates a class Display_TFT_Reverse_Feather to implement the DisplayHandler.

The interface has a couple standard functions:

• Idle function, when I simply want to say it’s scanning, plus total detections and how many are in range currently
• Detection function, when it shows drone information (MAC, RSSI, GPS)
• Info function, for just general info updates that aren’t the above two options
• Initialization function, for configuring the hardware for that particular board

For the implemntation on the feather, I used Adafruit’s example code a lot, using their custom TFT library, and it was pretty straightforward.

For the implementation on the lilygo, I used the Bodmer TFT_eSPI library, which was a bit of a bear to get working correctly. Selecting the configuration for the library felt confusing, and the current configuration for the lilygo t dongle doesn’t actually work with my hardware. I was using platform IO, which auto downloads the library for me, but then I had to modify my build flags so it used the correct configuration file (TFT_eSPI/User_Setups/Setup209_LilyGo_T_Dongle_S3.h), plus a bonus definition because my dongle apparently doesn’t match the standard setup (-DUSE_HSPI_PORT). Once I finally figured that out, the rest of the display implementation went quickly.

Firmware Change 2 - Add support for buttons

This currently assumes that if the buttons are enabled (via the BUTTONS_ENABLED definition), there will be 3 of them in a certain configuration (i.e. the reverse feather’s configuration). Eventually I’ll probably abstract things out similar to how the display is done above.

My code sets up interrupts on button presses, that feed to a queue that periodically gets serviced by the rtos task interactionTask. There’s a mini state machine there that sets up two buttons as a scroll up and scroll down button, with another one as a “detection dump” button. The scroll buttons lets you scroll between the currently in-range drones, looping back to the idle state when you finish.

The detection dump button triggers outputting all detections as json over serial. This supports using the reverse feather with a battery while you’re out and about, then when you get home, plug in the USB-C to your computer, open a serial connection, and press the button. Ta-da, all detections have been copied over! This isn’t maintained during power off events, but hey, it’s better than nothing.

Hardware - Adafruit TFT reverse feather

This gives a display to play with, to show drone detections immediately, some buttons, and of course an external antenna option. If you wanted, you could keep the buzzer by buying one and wiring it up to a free GPIO, there’s definitely room, but I haven’t gotten around to that yet, I was too excited by the display.

Here’s a link to the hardware list on adafruit: https://www.adafruit.com/wishlists/622656. The total is about 48 dollars, but if you want to skip the battery, or buy the internal antenna option, you could cut the price down considerably. Though this comes with functionality trade offs, of course.

A nice benefit from using the Adafruit TFT reverse feather is that it comes with a battery charger chip onboard, and a JST connector for a lipoly battery. Which means it’s super easy to power with either an external USB-C connector, or with an internal battery, for portability. I’m currently using the very advanced prototyping functionality of “cardboard boxes” to create a custom enclosure for this hardware, but someone with 3d printing knowledge could probably knock out a slick case for it pretty fast.

Here’s some pictures of that current setup:

Teardown Conference

I took this custom setup to Teardown 2026 and just had it out on a table with little infographics for part of the conference. I’ve included the infographic images below if you’re curious!

NOTE Teardown is a super fun conference, highly recommended

Hardware - Lilygo T Dongle S3

The Lilygo T-Dongle-S3 is another setup that gives me a display, but the biggest plus is the cute thumbdrive form factor, and how it was in stock at my local Microcenter so I could impulse buy it. Current cost was about 20 bucks (looks like direct from lilygo you can get it for cheaper). It has a built in antenna, but interestingly, there’s an option to update it to use an external antenna. The iPex connector is on the board, just disconnected, so you have to remove a 0 ohm resistor from one place and bridge two solder pads in another place, then boom, external antenna support. The snap on plastic case also snaps off pretty easily, from my experience, so this seems like a pretty easy process. I haven’t done more than taken a look though, so there may be some hidden issues there.

Another benefit of this is that it has a built in spot for a tiny SD card, funnily enough inside of the USB plug. This can be super useful for keeping data through a power off event, and is one of the reasons I want to look into saving detections to SD cards.

Current Detection Results

A neighbor was kind enough to fly a drone around for me, to let me test out some ranges of different solutions. He had a DJI drone, and we tested it around my neighborhood, which is relatively flat, with 2-story buildings and some trees. This is important context since wifi and bluetooth are both affected by things in the way, so if you live someplace where there’s no trees and hills, you’ll get better range than, say, hilly and tree-covered seattle. Where and how big the antenna is will affect range too. If you or the drone are higher up, you’ll get a better line-of-sight (or line-of-signal, I suppose) and so better range.

With my smaller antenna, at ground level, with the drone flying between 100 and 300 feet above ground level, my sky-spy setup can detect drones about 500-800 feet away consistently.

This is one day of testing, so there’s definitely wriggle room in that number, but it feels very promising considering I hadn’t tested if it worked at all yet!

There’s other drone detection setups out there (though not that many), but this project has a relatively low cost of entry with decent range of detection. I feel either Colonel Panic’s or my codebase will continue to improve, too, making it a project to watch.

References

Colonel Panic: https://colonelpanic.tech/

Sky-Spy: github.com/colonelpanichacks/Sky-Spy

My Modified codebase: https://github.com/Dthurow/Sky-Spy

Adafruit TFT reverse feather drone kit: https://www.adafruit.com/wishlists/622656

Lilygo T Dongle S-3: https://www.microcenter.com/product/703093/lilygo-t-dongle-s3-esp32-s3-ttgo-development-board-with-screen

open remote ID library: https://github.com/opendroneid/opendroneid-core-c

Drones have gotten more and more popular lately, from filming fun things on social media, delivering packages for Amazon, being used by law enforcements in the states, all the way to being used in the ukraine war. Understanding the technology and building tech to track the drones seems very useful. We’ve had public tracking of airplanes via ADSB for years at this point (see my previous post about setting up your own flight tracker with ADSB), but no equivalent with drones. Partially this is because drones, up until recently, didn’t really have ways to track them, unless you were visually tracking them or doing some advanced radio frequency shenanigans.

That has changed relatively recently (in 2023) when the FAA started requiring Remote IDs for drones, the functional equivalent to the broadcast ADSB info, but for drones specifically. Specific information about the drone, its location, and the drone operator are broadcast in one of the variety of ways specified by the standard put out by the ASTM (a standards governing body), which is charmingly named “F3411”. With this new ability, it’s possible to start tracking and aggregating drone flight information, same as ADSB. As the FAA website shows (screenshot below), there are now several ways to be compliant with the remote ID requirements, and 2/3rds of the ways require actually broadcasting the info. Lets take a look at this new protocol, and poke around at some sample packets (or generate our own).

Remote ID Basics

Remote ID is the FAA-required ability to send out a drone’s information while in flight, including things like: current latitude/longitude, some type of ID, pilot location, height, speed, etc. The actual implementation used by drone manufacturers is defined in the ASTM F3411 standard. It gives specifications for drones to send out data in a couple different ways:

• Wifi NAN discovery frames
• Wifi Beacon frames
• Bluetooth 4.x advertising beacon messages, called “Legacy Advertising” in the ASTMF3411.
• Bluetooth 5 Extended Advertising messages

Since these messages are unencoded, someone with the right software and some wifi and bluetooth dongles can detect these messages. However, in order to do that, you need at least a basic understanding of a lot of different protocols that the remote ID info will be wrapped in, not to mention how the remote ID info is encoded.

So, let’s first see what sort of packets are being sent that contain the remote ID info, then look into the remote ID info itself.

If you prefer to just read the specification directly, you can access the ASTM F3411-19 revision here, though there has been a newer revision released in 2022. This does not appear to be accessible for free as of this writing.

Wifi

Drones will broadcast on channel 6 (2.437 GHz) in the 2.4 GHz frequency band and may optionally operate in channel 149 (5.745 GHz) in the 5 GHz band. From real-world experience so far, it looks like DJIs will broadcast on 2.4GHz, and Skydio drones are more likely to broadcast solely on 5GHz.

NOTE So if you want to listen for wifi remote ID messages, you have to make sure you’re listening on those specific channels, otherwise you’ll never hear them! This is important, for, say, using wireshark, mentioned below.

Wifi NAN discovery frames

The NAN discovery frames are actually NAN-specific “public action” Wifi frames. You can narrow this down by looking at the Management Header of a wifi frame first. In there is the “Frame Control” section, which contains the “Type” and “Subtype” fields.To make sure it’s an action frame, you’ll check if the Type is “management” 0x0, and Subtype is “action” 0xD. The “public” part will be verified later in the frame. After the Frame Control section, there’s a destination address. The NAN Network ID specification calls for the destination address in the header to be 51:6F:9A:01:00:00.

Once it’s confirmed it’s a Wifi NAN action frame, you can confirm the Category ID is “public” 0x4 in the NAN Service Discovery Header. We now know it’s a Public Action NAN frame. But we still don’t even know if it’s a drone broadcasting a remote ID packet!

For that, we can check this by looking at the OUI and Service ID inside the frame’s “NAN Service Discovery Header” and its “Service Descriptor Attribute” section. The OUI in the Service Discovery Header is set to the Wifi Alliance OUI, 50:6F:9A.

OUI stands for Organizationally Unique Identifier and is a 24-bit number that uniquely identifies a vendor, manufacturer, or other organization

And in the “Service Descriptor Attribute” section, the Service ID is the first 6 bytes of a sha of "org.opendroneid.remoteid", which is 88:69:19:9d:92:09.

You can confirm the 6 bytes for the Service ID are correct for yourself by hashing it with some python real quick:

>> blah = hashlib.sha256(b"org.opendroneid.remoteid").digest()[:6]
>> print(", ".join(hex(b) for b in blah))
0x88, 0x69, 0x19, 0x9d, 0x92, 0x9

Here’s a screenshot of the F3411 document for the NAN discovery frame. See if you can spot all the values listed above in the frame:

Congratulations, you’ve now parsed enough to confirm it’s a Remote ID Wifi message, and you can start processing the info! Jump down farther in this post to see what’s inside. Otherwise, lets switch to a different packet option…

Beacon

Another way to send out the remote IDs is via Wifi Beacon frames. These have a different format, and so they need to be parsed differently.

First, take a look at the Management Header of the wifi frame, like last time. Looking in the Frame Control section again, you’ll need to verify the Type is “management” 0x0, Subtype is “Beacon” 0x8.

Now this is where the format changes. Inside the Beacon is several required fields, followed by a series of Information Elements that is basically a series of key/value pairs with info the Beacon wants to share. In our case, we want remote ID beacon info. So we need to look through the beacon’s Information Element “tags” (which are the keys in the key/value idea), for 0xdd, the Vendor Specific Information element type. Then verify the start of the info in that tag (the “value” in the key/value idea) is the ASTM OUI, FA:0B:BC. We also want to check the type after the OUI is in there and is set to 0x0D, which indicates a Remote ID app code.

Here’s a screenshot with an example of this packet in wireshark. The left side is the parsed packet with the field names and values, and the right side is the raw packet. Note the type/subtype at the top, the Vendor Specific Information Element type, and the OUI listed there.

NOTE If you haven’t done much with wireshark, highly recommend finding some tutorials and poking around in things. It’s amazing for parsing packets of all sorts of protocols, I’ve found endless use for it, from my web dev days, to now.

The rest of the info in there is a remote ID message, yay!

Bluetooth

Bluetooth messages are sent over Bluetooth Low Energy (BLE) the most common embedded bluetooth option. The specification supports BLE 4 and 5.

Bluetooth 4.x

BLE 4 uses advertising beacon messages, called “Legacy Advertising” in the ASTMF3411. Advertising messages are sent on a specific set of channels in the bluetooth frequency range, channels 37, 38, and 39, and the beacon messages are simply sending information into the void (not sending to a particular end point), with a small payload. Which in our case, is the Remote ID messages.

To filter down to the remote ID messages, we need to grab all Advertising beacon messages, and check the “AD info” field. That will contain the Mfg Code, which should be 0xFFFA, the ASTM code. It will also contain the “AD App” field, which should be 0x0D, for remote ID. The content in this advertising message will be what we want. Here’s another screenshot from the ASTM F3411 spec, showing those fields in the packet:

Bluetooth 5

Bluetooth 5 Extended Advertisements messages, when sent on a LE Coded (S=8) PHY, can increase the range of the advertisements by a factor of 4, which is why it’s so nice. It achieves this by cutting the rate of the data transfer down (with S=8, data rate is 1/8th), where multiple symbols make up 1 bit (with S=8… it’s eight symbols per bit), this lets the receiver parse things from a farther distance.

These Extended advertisements also use a two-step advertisement messages, because it uses the advertisement types ADV_EXT_IND and AUX_ADV_IND. So they send advertisements on the normal advertising channels like BLE 4, but those advertisements actually point to larger advertisement packets on other bluetooth channels. This second, larger advertisement frame will have the actual remote ID information. Like the BLE 4 protocol, it will contain the Mfg Code, which should be 0xFFFA and the “AD App” field, which should be 0x0D.

Because the secondary advertisement frame is larger, the remote ID information will be sent as a “message pack” type, which combines different remote ID messages into one large packet.

What are these “message pack” and other remote ID message types? I’m glad you asked…

Remote ID messages

So after all the parsing above, you’ve finally gotten a Remote ID message. But what’s in that message? There’s several different type of messages according to the full ASTM F3411:

The main ones we care about when detecting drones though, are just a subset: Basic ID, Location/Vector, System, and Message Pack. Each carry a different subset of data, but can be combined to give a full overview of what the drone and operator are doing.

Message Pack

This one is easy - it’s just a collection of the other messages, so adding support for this is just allowing there to be multiple messages inside a single sent frame (however it was sent). Used in BLE 5 extended advertising messages, since they have the room for it.

Basic ID

This gives the ID for the drone, and its ID type. E.g. DJI drones will broadcast their Serial numbers as their ID, with ID Type of, you guessed it, “Serial Number (ANSI/CTA-2063-A)”. It will also give the type of drone it is, e.g. fixed wing, helicopter/multirotor, etc. Sadly, I have not spotted any drones listing their type as “Airship (such as a blimp)”, but maybe someday.

Location/Vector

This message gives info about the location, altitude, direction, and speed of the drone. One interesting thing to note about altitude, there’s a AGL height which is height Above Ground Level. This references from the drone’s take-off point, and may not actually be accurate to altitude above the ground where it is actively flying.

System

This message gives the operator’s latitude and longitude. Depending on how things are setup, this will just default to the drone’s take-off location, OR it will be updated dynamically.

Using This Info - Wireshark

You’ve learned a bunch of protocol things, but what can you actually do with it? First and simplest option is to setup a laptop with wireshark and a custom protocol dissector so you can actually capture the packets you just learned about above yourself! You’ll need a laptop that can run wireshark, a wifi or bluetooth dongle to capture the packets, and a drone that broadcasts its Remote ID (e.g. a DJI Neo).

If you want to just play around with packets, the dissector comes with a few pcaps you can load and poke around at. So you would only have to install wireshark and add the dissector. E.g. https://github.com/opendroneid/wireshark-dissector/blob/main/odid_wifi_sample.pcap

The basic steps for this are:

• Install Wireshark, setup so non-root users can capture packets
• Install custom protocol dissector, to parse Remote IDs
• Plug a wifi or bluetooth dongle into your computer
  • if it’s a wifi dongle, make sure it supports “monitor” mode and set the 2.4GHz channel to “6”
  • other setups can be found in the readme of the protocol dissector
• Run wireshark
• Fly a drone that broadcasts Remote ID on the channel you’re lisenting on (e.g. 2.4GHz channel 6)
• Capture packets
• Profit!

On my linux laptop, these commands were useful to setup my wifi dongle, a Panda PAU0F to monitor mode on channel 6. Note the interface name wlx9cefd5f644bd will be different for you, determine by simply running sudo ifconfig with the wifi dongle unplugged and then plugged in, and see what interface appears. These commands: take the interface down so I can modify the configuration, set the monitor flag, bring the interface back up, and set it to monitor channel 6.

sudo ifconfig wlx9cefd5f644bd down
sudo iw wlx9cefd5f644bd set monitor none
sudo ifconfig wlx9cefd5f644bd up

sudo iw dev wlx9cefd5f644bd set channel 6

NOTE not all wifi dongles can be set to monitor mode! You’ll have to research before you buy. Acrylic wifi has a list of monitor mode dongles that may work with your computer here

Conclusion

Hopefully this whirlwind overview of the protocols involved in Remote IDs can give you better context for what you’d want to look for when detecting drones via Remote IDs. My next post will most likely be an overview of currently existing drone detection software with their pros and cons, as well as what work I’ve done personally to expand those options.

References

Open Drone ID library: https://github.com/opendroneid/opendroneid-core-c

IEEE article with an overview of drone tracking technology and regulations: https://ieeexplore.ieee.org/document/9861637

ASTM F3411 Wi-Fi Beacon protocol doc https://thedroneprofessor.com/wp-content/uploads/2022/11/F3411.40165-UAS-Remote-ID.pdf

Setting a wifi interface to monitor mode: https://www.geeksforgeeks.org/linux-unix/how-to-put-wifi-interface-into-monitor-mode-in-linux/

Wireshark dissector of drone IDs https://github.com/opendroneid/wireshark-dissector/tree/main

Monitor mode wifi dongles: https://www.acrylicwifi.com/en/wifi-analyzer/requirements-and-compatibility/

Playing around with different development projects, there’s often a need to listen on a USB serial port connection to see log outputs. If I’m making a device that I want to test out in the wild, without a full display yet, it’d be nice to have a grab-and-go tool for that. So, I decided to put together a raspberry pi zero I had lying around, plus the adafruit e-ink bonnet and a powerbank, to let me read serial logs on the road.

Note before I get into it If you want to be boring (aka practical), you should check out phone apps that allow you to read serial info from devices connected to your phone, that may be a simpler solution than buying more hardware. My way is more fun though.

Hardware Needed

Below is the hardware I used for development and my final product. Net price for this, if you don’t have things lying around already, will probably run 65-75 bucks. The power bank was the spendiest buy, at 27 dollars, so if you already have one that would work, that helps drop the price down.

• adafruit e-ink bonnet
• powerbank capable of 5V output
• raspberry pi zero
  • NOTE for the zero, it doesn’t have wifi, so you need to make sure you have the right tools for development. I used:
    • USB OTG Host Cable - MicroB OTG male to A female
    • a powered USB hub
    • A wifi dongle of some sort
  • for ease of development, if you’re buying new, it would make sense to instead by a raspberry pi zero W, so you can more easily connect to the pi via wifi/ssh
• power cord for the pi zero, for during development
• if you want to keep the pi protected, I found the adafruit pi zero case fits well with the e-ink screen on top
• micro USB cable that can connect to whatever projects you’re working on (in my case, I needed a micro USB to USB C)
• SD card for the pi’s OS
• Micro-hdmi adaptor so you can connect to a monitor, and a keyboard and mouse for the pi
  • Technically if you do everything right during initial setup, you don’t need this. In the real world it’s good to have everything for troubleshooting during your initial Pi setup.

Why e-ink?

Cuz it’s fun, honestly. A common argument for e-inks is power saving, since you don’t have to give it power to keep what was last on the screen, but I haven’t profiled that so I can’t say for sure. I really like the visual look of e-inks, and want to work on future projects with e-inks, so I figured this would be an easy project to do some basic testing with an e-ink screen.

A common “con” of e-inks is the refresh rate, a lot of them have miserable refresh rates if you want real-time or near-real-time responses. For the sort of projects I work on, I’m not getting spammed by serial logs, so the slower timing isn’t a problem. And from my testing, this e-ink screen has pretty slow refresh. I suspect the slow refresh could be improved if I dug into the docs more and looked for some time-saving hacks either in the python library from adafruit or how I’m displaying things, but for now, I’m just keeping things simple.

Setup And Verification

Hardware setup

Put the pi in the case if you’re using one, then connect the e-ink bonnet to the raspberry pi. Make sure it’s fully seated, I had some weird issues with the display not working that turned out to be because I hadn’t pushed them together hard enough.

Plug in the OTG cable, hook it up to your powered USB hub, and plug in your wifi dongle, and if you want to do setup using the pi’s GUI, connect a keyboard and mouse here.

Software

Flash the latest version of the raspberry pi OS for your zero, using the Raspberry Pi Imager program. You’ll also want to setup your username and password and enable SSH. If you have a pi zero with no wifi, you’ll want to setup a keyboard and monitor so you can configure the wifi via the wifi dongle. Once the initial setup is done, you should be able to just remote in via ssh and do future steps over that. If you haven’t had experience setting up a pi before, the official docs are pretty good.

Install the blinka library from adafruit. They have good instructions here, that involve setting up a virtual environment and adding their blinka python script to the $PATH environment variable.

Once you have your virtual environment setup, you’ll also want to install the EPD library and the pillow library with this:

pip3 install adafruit-circuitpython-epd
sudo apt-get install python3-pil

To validate everything works, you can run the REPL (python3) on your pi and run the following script, it should print a “Hello world” message on the e-ink screen, and you can then write more to the screen by just calling the display_text function, e.g. display_text("my new string!")

import board
import busio
import digitalio
from PIL import Image, ImageDraw, ImageFont
from adafruit_epd.epd import Adafruit_EPD
import serial
import time
import serial.tools.list_ports


from adafruit_epd.ssd1680 import Adafruit_SSD1680Z


# First define some color constants
WHITE = (0xFF, 0xFF, 0xFF)
BLACK = (0x00, 0x00, 0x00)

# Next define some constants to allow easy resizing of shapes and colors
BORDER = 5
FONTSIZE = 18
FONT_MAX_LENGTH = 27
BACKGROUND_COLOR = WHITE
TEXT_COLOR = BLACK


# create the spi device and pins we will need
spi = busio.SPI(board.SCK, MOSI=board.MOSI, MISO=board.MISO)
ecs = digitalio.DigitalInOut(board.CE0)
dc = digitalio.DigitalInOut(board.D22)
rst = digitalio.DigitalInOut(board.D27)
busy = digitalio.DigitalInOut(board.D17)
srcs = None


# e ink display
display = Adafruit_SSD1680Z(122, 250, spi, cs_pin=ecs, dc_pin=dc, sramcs_pin=srcs,
                          rst_pin=rst, busy_pin=busy)


display.rotation = 1

# re-draw the entire screen with the new text message
def display_text( text: str):
    image = Image.new("RGB", (display.width, display.height), WHITE)

    # Get drawing object to draw on image.
    draw = ImageDraw.Draw(image)

    # Draw a filled box as the background
    draw.rectangle(
        (0, 0, display.width - 1, display.height - 1),
        fill=BACKGROUND_COLOR,
    )

    # Load a TTF Font
    # NOTE: linux systems normally default have this font
    font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", FONTSIZE)

    # Draw the text
    draw.text(
    (BORDER, BORDER),
    text,
    font=font,
    fill=TEXT_COLOR,
    )

    # Display image.
    display.image(image)
    display.display()

display_text("Hello World!")

Python Script

This script is the meat of the project. The basic functionality it sets up is:

• Configure and connect to the Adafruit E-ink screen using circuit python and adafruit’s libraries
• Constantly listen on a given serial port for any text
  • Currently the script hard codes which USB device to listen to, and what baudrate/serial configuration it has
• Given the text, re-format it a bit to make sure it doesn’t run off the screen of the e-ink, and add a timestamp
  • It also checks if it’s receiving the exact same text repeatedly (e.g. a heartbeat message from the device). In this case, it will only update the e-ink screen if it’s been longer than a minute, so the timestamp gets updated and you know the script is still working.
• Using the Pillow library, draw a new image for the e-ink screen with the re-formatted text, then send it to the e-ink to display

You can directly run the script on the pi by copying it over and then using ssh to run.

Don’t Forget You’ll need to activate your python virtual environment before you try running it though!

As it’s just shy of 175 lines, I’ll just include the full script below in the Full Code section, for you to take a closer look.

I do want to point out some interesting pieces of the script.

Finding the serial device

Finding the device I want to listen to ends up being more complicated than you may think. If, for example, I’m using a Xiao ESP32 device, I can plug in a USB C to micro USB cable to it and the raspberry pi, and it will be automounted at a standard /dev/tty<blah> (e.g. /dev/ttyACM0). However, the exact number of this ESP32 can change from one boot up to another. So, if I just hardcode the serial connection, e.g. serial.Serial('/dev/ttyACM0', 115200, timeout=1), the script will occasonally fail.

My workaround for this is to use the Product ID and Vendor ID (PID and VID) and the serial.tools.list_ports functionality in python. The PID and VID are part of USB standard ways of identifying devices, the Vendor ID is assigned to a particular company by the USB Committee, and the Product ID is assigned by the company to a particular USB device.

With this, I can look through connected USB devices and filter down to only devices that are specifically an Espressif USB JTAG/serial connection, and connect to the first one I find. Since I only ever have one ESP32 connected, it works great.

After I wrote this all up, I did some extra searching online, and did see some people solved this by creating udev rules to create permanent /dev/ mount points for their USB devices. The trade off here is that you have to do more customizing of the raspberry pi OS. I was trying to keep this to a minimum to make it easy to create more of these setups in the future, so I’m satisified with the python solution for now.

Troubleshooting Finding Your Device

If you want to use this script, and you’re having issues with getting the right PID and VID, or you’re unsure if python is actually seeing your device, this lil python script, run in the REPL can help you out:

import serial
import serial.tools.list_ports
for p in serial.tools.list_ports.comports():
    print(p.pid)
    print(p.vid)
    print(p.manufacturer)

Running the lsusb command can also help, to make sure the pi is seeing your device, and it also gives your pid/vid in hex

> lsusb
Bus 001 Device 006: ID 303a:1001 Espressif USB JTAG/serial debug unit

Checking in multiple different ways can narrow down which part of your system is currently acting up. E.g. if python isn’t showing your device but the lsusb is, maybe there’s an issue with your python installation. But if the python script and lsusb cant see it, maybe it’s an issue with your device, or the physical port.

Fun Fact testing in multiple ways to narrow down which “thing” is breaking, is a super valuable skill for firmware development, software development, and general life. (insert “the more you know” banner here)

Reformatting text

To keep any text I get from the serial device from running off the right side of the e-ink screen, I added a function that replaces spaces with the newline character \n whenever the text starts getting too long. This feels like a nice way to display the text without getting weird cutoffs, though it depends on the type of text you’re getting from your serial device.

I also currently dont have it check to see if the text is too long and will go off the bottom of the screen. My current use doesn’t need that functionality, so why add it? If I do, I’ll most likely update the script to do more of a buffer queue system, to support faster serial logs as well as longer messages.

def reformat_text(text: str, maxlength: int):
    new_string = ""
    cur_row_length = 0
    for word in text.split(' '):
        if cur_row_length + len(word) < maxlength:
            if len(new_string) == 0:
                new_string = word
                cur_row_length = len(word)
            else:
                new_string = new_string + " " + word
                cur_row_length += len(word) + 1
        else:
            new_string = new_string + "\n" + word
            cur_row_length = len(word)
    return new_string

Systemd

Once I have the python script working, I want to run it automatically on every boot. This means I will have to dip my toes into the systemd world.

Systemd manages services on linux, where services are just long-running programs that can be configured to run at certain times, or in my case, on boot, and then forever. To tell Systemd I have a new program I want it to handle, I have to make a service configuration file in the right location that it’ll look at, and tell it I want it to boot, and if my program crashes out, have it restart it.

The following Systemd service configuration file, does just that, when saved at /lib/systemd/system/shellscript.service.

[Unit]  
Description=My Python Autostart Script  
After=multi-user.target  
 
[Service]  
User=daniellet
WorkingDirectory=/home/daniellet
ExecStart=/home/daniellet/start_script.sh  
Restart=always  
RestartSec=5  
 
[Install]  
WantedBy=multi-user.target 

I created my python file and saved it in my user’s home directory, which is why I tell systemd to run as me, in my /home/daniellet directory. The ExecStart tells it what script to run. But you may notice, that script isn’t my python script! Instead, it’s a trusty bash script, what’s up with that!

We can’t run the python script directly simply because the python script is running in a virtual environment, and you will need to activate the virtual environment every time the Pi is rebooted. So this bash script, saved as start_script.sh, does just that. Create it in your home directory, and be sure to run a chmod +x to make it executable after you make it.

#!/bin/bash
source env/bin/activate
python3 -u printing_to_eink.py

The -u flag on the python command is actually important. My python script prints logging info to standard out. Systemd has built in support for logging standard out info from the services it runs, but python buffers the standard out, and when testing, was resulting in the logging output not actually getting logged. The solution was to add the -u flag, which makes it unbuffered and output right away.

Once the bash script and config file are made, you’ll need to reload the systemd system so it re-looks for configuration files and sees my new service. You can do this with sudo systemctl daemon-reload . Now you can start using the systemd commands to mess with your new service! The service name will be the name of the configuration file you made (in my case, shellscript.service)

Here are some useful commands around systemd for troubleshooting your setup

Manually start/stop the service:

sudo systemctl start shellscript.service
sudo systemctl stop shellscript.service

Check Current Status of the service:

sudo systemctl status shellscript.service

Toggle if the service runs at boot:

sudo systemctl enable shellscript.service
sudo systemctl disable shellscript.service

Now, you can also see the logging output from my python script by using journalctl. When it’s running, you can use the following command to see the logs:

journalctl -u shellscript.service

With the service now enabled, you should be able to unplug and re-plug in the power to the pi zero, and the e-ink screen should display a “Booting up…” message, followed by the serial output from your device! You did it!

NOTE The initial bootup time on my pi is pretty dang slow, so be patient!

Future improvements

Some ideas for future improvements that may or may not come to pass:

• Update the script to not hard-code the PID/VID and baudrate
  • or update to have hardcoded udev rules so the script doesn’t have to hunt for the serial connection
• make a nifty enclosure for it, either 3D printed or some other way
• Currently the timestamp is what it thinks is central time, but since it doesn’t have an internal battery, when I pull the power, the clock doesn’t stay up to date. And with no internal wifi, it doesn’t update time from the internet. So I’d like to either add an RTC battery, or update the timestamp to simply be time from boot

Conclusion

Setting up the e-ink screen and getting it running for a basic use like this was easier than I had expected. I think I probably spent more time making sure my systemd script was setup correctly than writing the python script. And I did end up using this setup a couple times while testing other devices, so it wasn’t just a fun project with no application. I’ll definitely be keeping this e-ink screen in mind for future projects.

References

Adafruit’s e-ink guide: https://learn.adafruit.com/2-13-in-e-ink-bonnet

Setting up circuitpython blinka library: https://learn.adafruit.com/circuitpython-on-raspberrypi-linux/installing-circuitpython-on-raspberry-pi

Setting udev rules on raspberry pi: https://loketdiversen.nl/2015/02/howto-persistent-device-names-on-raspberry-pi/

Setting up systemd on linux: https://tecadmin.net/run-shell-script-as-systemd-service/

Full Code

Below is the full code for each part of this project. The Systemd configuration and bash file, and the python script.

Systemd

systemd service configuration file, saved at /lib/systemd/system/shellscript.service

[Unit]  
Description=My Python Autostart Script  
After=multi-user.target  
 
[Service]  
User=daniellet
WorkingDirectory=/home/daniellet
ExecStart=/home/daniellet/start_script.sh  
Restart=always  
RestartSec=5  
 
[Install]  
WantedBy=multi-user.target 

Bash script

Bash script to activate the python environment and run the python script. Saved as start_script.sh in my home directory.

#!/bin/bash
source env/bin/activate
python3 -u printing_to_eink.py

Python Script

Here is the printing_to_eink.py file that is the meat of the functionality, saved in my home directory. Note the base code was originally pulled from Adafruit’s usage guide and then modified for my particular setup.

"""
ePaper Display text coming from a serial port using the Pillow Library.

Based on the python scripts from https://learn.adafruit.com/2-13-in-e-ink-bonnet/usage
modified for this particular use case

ASSUMES:
- connected to SSD1680Z e-ink bonnet
- screen is black and white
- there is a serial port connected on a USB
- Installed CircuitPython Blinka library from adafruit https://learn.adafruit.com/circuitpython-on-raspberrypi-linux/installing-circuitpython-on-raspberry-pi
- script is running on raspberry pi zero. Has not been tested on other raspberry pi systems (though should work...)

"""

import board
import busio
import digitalio
from PIL import Image, ImageDraw, ImageFont
from adafruit_epd.epd import Adafruit_EPD
import serial
import time
import serial.tools.list_ports


from adafruit_epd.ssd1680 import Adafruit_SSD1680Z


# First define some color constants
WHITE = (0xFF, 0xFF, 0xFF)
BLACK = (0x00, 0x00, 0x00)

# Next define some constants to allow easy resizing of shapes and colors
BORDER = 5
FONTSIZE = 18
FONT_MAX_LENGTH = 27
BACKGROUND_COLOR = WHITE
TEXT_COLOR = BLACK


# create the spi device and pins we will need
spi = busio.SPI(board.SCK, MOSI=board.MOSI, MISO=board.MISO)
ecs = digitalio.DigitalInOut(board.CE0)
dc = digitalio.DigitalInOut(board.D22)
rst = digitalio.DigitalInOut(board.D27)
busy = digitalio.DigitalInOut(board.D17)
srcs = None


# e ink display
display = Adafruit_SSD1680Z(122, 250, spi, cs_pin=ecs, dc_pin=dc, sramcs_pin=srcs,
                          rst_pin=rst, busy_pin=busy)


display.rotation = 1

# replace some space with a newline character so the string will fit on
# the given screen. NOTE: does not have a max number of rows check
def reformat_text(text: str, maxlength: int):
    new_string = ""
    cur_row_length = 0
    for word in text.split(' '):
        if cur_row_length + len(word) < maxlength:
            if len(new_string) == 0:
                new_string = word
                cur_row_length = len(word)
            else:
                new_string = new_string + " " + word
                cur_row_length += len(word) + 1
        else:
            new_string = new_string + "\n" + word
            cur_row_length = len(word)
    return new_string

# re-draw the entire screen with the new text message
def display_text( text: str):
    image = Image.new("RGB", (display.width, display.height), WHITE)

    # Get drawing object to draw on image.
    draw = ImageDraw.Draw(image)

    # Draw a filled box as the background
    draw.rectangle(
        (0, 0, display.width - 1, display.height - 1),
        fill=BACKGROUND_COLOR,
    )

    # Load a TTF Font
    # NOTE: linux systems normally default have this font
    font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", FONTSIZE)

    # Draw the text after its been reformated
    new_text = reformat_text(text, FONT_MAX_LENGTH)
    draw.text(
    (BORDER, BORDER),
    new_text,
    font=font,
    fill=TEXT_COLOR,
    )

    # Display image.
    display.image(image)
    display.display()

def process_serial_stream(ser, line_processor):
    """Process continuous serial data stream"""
    line_buffer = b''
    
    while True:
        try:
            # Read available data
            if ser.in_waiting:
                chunk = ser.read(ser.in_waiting)
                line_buffer += chunk
                
                # Process complete lines
                while b'\n' in line_buffer:
                    line, line_buffer = line_buffer.split(b'\n', 1)
                    try:
                        text = line.decode('utf-8').strip()
                        if text:
                            result = line_processor(text)
                            if result:
                                yield result
                    except Exception as e:
                        print(f"Process error: {e}")
            else:
                time.sleep(0.001)
                
        except KeyboardInterrupt:
            break
        except Exception as e:
            print(f"Stream error: {e}")

def parse_msg(line):
    """check if it was a detection, and return if so"""
    # for testing just return it all
    # TODO can add filtering here if the line is something we dont want to print
    # e.g. only print lines that say "DEBUG" or "ERR" in them
    return line

print("Booting up...")
display_text(f"{time.strftime("[%H:%M:%S]")} Booting up...")
last_display_update = time.time()
# trigger serial tracking
last_serial_string = ""
try:

    #find espressif connected board
    #TODO if connected dev board is diff, need to update the PID/VID here
    portname = ""
    for p in serial.tools.list_ports.comports():
        if p.pid == 4097 and p.vid == 12346:
            portname = p.device

    if portname == "":
        print("CANT FIND PORT")
    else:
        print(f"Found port {portname}, connecting...")
        # assumes connection is 115200 baud rate
        # TODO update baudrate if needed
        with serial.Serial(portname, 115200, timeout=1) as ser:
            for detection in process_serial_stream(ser, parse_msg):
                print(f"detected string: {detection}")
                # print all new messages, and if it's been the same message for over 60 seconds
                # (e.g. the same status message of "scanning" or something)
                # print the same message but with updated timestamp, to show its not dead
                if (last_serial_string != detection or (time.time() - last_display_update) > 60):
                    display_text(f"{time.strftime("[%H:%M:%S]")} {detection}")
                    last_serial_string = detection
                    last_display_update = time.time()
                    
except Exception as e:
    print(f"Serial connection failed {e}")

Radio has always been a side interest of mine, but seemed too convoluted for me to easily get into. However, now with Software-Defined Radios (SDR), things started to shift into the software world, so I started to get my toes wet. One use that seemed pretty popular was tracking airplanes, so I thought I’d give it a go.

I had previously setup an ADSB receiver using the adsb exchange raspberry pi image, and found it a little finicky. I also had other uses for the pi, so I ended up retiring that setup, with the eventual plan to get it back up and running “sometime in the future”. Which turns out is around now! (don’t…don’t ask how long it was since I did that setup the first time)

This time, I came across a different raspberry pi image that was actually default included in the rasperry pi imager tool, called ADSB.im, and decided to give it a try again. I’ve had it setup for about a month now (probably more once this is published!), and have learned a lot, so I thought I’d share.

Basics for Flight Tracking

Besides looking at planes with your eyeballs, there’s a variety of ways to track aircraft. The most common is looking at the ADSB data. The Automatic Dependent Surveillance–Broadcast is broadcast on the 1090 Mhz or 978 MHz frequency, sending out info like speed, course, altitude, callsign and identification of an aircraft. You don’t even have to setup your own station to see this, you can go to ADSBExchange or ADSB.lol to see them right now!

There’s also a lot of ways to listen in on the air traffic control conversations, but that’s not talked about in this article because that’s a different can of worms. Though I do want to mention, the Minneapolis airport has a live youtube video that shows the airport and also plays the conversations: https://www.youtube.com/live/FY6WlMjzG2U?si=9Io45ISSSH3IGIuM

Slightly more in-depth info for Flight Tracking

To get more in-depth, sure hope you like some acronyms! The two main frequency we care about right now is the 978MHz and the 1090MHz.

978MHz - The 978MHz frequency is used by the universal access transceiver (UAT). UAT is intended to support not only ADS-B, but also flight information service – broadcast (FIS-B), traffic information service – broadcast (TIS-B),

FIS-B - is only broadcast on UAT, and provides weather info to aircraft

TIS-B - is broadcast on both 1090MHz and 978MHz, and provides info from ground stations about aircraft that only have a basic transponder and aren’t transmitting out ADSB data directly.

1090MHz has both Mode S and 1090 extended squitter (ES) on it. The Mode S was a replacement for a different protocol, air traffic control radar beacon system (ATCRBS). And the 1090 ES is indeed an extension of the protocol, with all the ADSB data.

If you want to play with even more frequencies:

ACARS - Aircraft Communications Addressing and Reporting System. Standard ACARS transmits at a frequency of 131.550 MHz. This is used for communicating between air traffic control and the aircraft.

VDL2 - As this article states: The VHF Data Link mode 2 (VDL2) is a new transmission mode used on aircraft for sending short messages, position data (similar to ADS-B) and also for allowing traffic controllers to communicate to pilots via text and data. VDL2 is intended to eventually replace the standard ACARS modes. It is found at 136.975 MHz.

Help, My Eyes Just Glazed Over!

ADSB is messages sent out by planes and helicopters, it gives info about where they are and where they’re heading. You can watch them in real time online, or setup a local station with a software defined radio, a raspberry pi, and some real simple software to glue it all together!

Why can I see a helicopter out my window but not online?

When I first started learning about this, there were a couple times I spotted a plane or helicopter, but when I went to the adsb tracker, I didn’t see anything! Turns out, there’s a couple reasons a plane or helicopter might not be seen in an online ADSB aggregator websites.

• The LADD flag - LADD Program was created by the FAA which lets private aircraft owers to request their aircraft’s flight data be removed from general distribution. Some ADSB websites follow this flag and remove the aircraft, some do not. As of this writing, I know adsb.lol still displays them.

• Their transponder is off - Some government agencies (e.g. DHS) are allowed to turn off their ADSB transponders when they’re in flight on various missions. This means the ADSB data simply doesn’t exist to be collected! So they can’t be tracked in this way.

• Poor coverage - The ADSB aggregator websites are, in fact, aggregators of data, normally simply volunteers uploading (creating “feeders” like described below). If there’s no one near enough monitoring and feeding the ADSB data to aggregators, it wont show up. That also means you’re in the perfect place to help out the global aviation community!

The Hardware and Setup

Hardware needed:

• Raspberry Pi 4 model B with 1GB RAM
• Software Defined Radio from adafruit
• raspberry pi power source
• SD card

With everything all connected, it’s a pretty compact setup I can put on my second floor windowsill. Since the airport is south of me, I put it in my south-facing window.

Initial setup

I found the ADSB.im website had a really good setup guide that was straight-forward to follow. The biggest issue was simply trying to find the IP address on my network once the pi was booted up. For some reason the various auto-finder options weren’t working, so I had to dredge up some old networking knowledge.

To hunt for the pi on my network, I ran ifconfig on my linux laptop to see what my IP address was, then did a quick ping scan with nmap to see who else was on my network (I knew no one else but me was connected right now)

nmap -sn 192.168.68.0/24

That quickly showed where it was living, and I went to port 80 in my browser, and success!

How well it works

Even with the default antenna and no fine-tuning on my part, I’m getting aircraft! The pi image comes with graphs1090 data, so I can even see some more in-depth data on my system. My median range is about 11.7 nautical miles, which is about 13.4 regular ole miles. This feels pretty decent to me, and since I’m so close to the minneapolis airport, I still get a lot of aircraft even with that distance

On the pi side, I can see the tracking is very low CPU usage, which is nice, I’m not burning up the whole pi trying to do anything too fancy.

I also enabled skystats (more below), which increases IO writes, and boy can you see that in the disk IO graph. I assume at some point that will fry the SD card in my pi, but for now, I’m fine with being rough on it in exchange for fun info about planes.

Fun configurations

In the Setup -> Advanced setting on the website, you can enable SkyStats

Skystats is a docker container you can run on the raspberry pi that watches the ADSB data coming in and collects various stats, and of particular interest, flags interesting aircraft using a local copy of plane-alert-db, which lists interesting aircraft and why they’re interesting. It also hooks into various airplane photo websites, like jetphotos.com to give you pictures, even if you missed seeing it yourself as it flew by.

There’s a lot of other configurations I haven’t even touched on the pi, but that’s a future todo for now.

Interesting Planes

I’ve never really noticed planes outside of the constant stream of Delta flights heading out from the airport, so it was fun to see models I’d never heard of before. Here’s a couple that stuck out.

Short SkyVan

I’ve never heard of this plane before, but when I was poking around at SkyStat’s interesting airplanes, this just made me laugh. Look at this ridiculous plane!

(Image taken from jetphotos.com)

Looking into who owns it and what it’s for, it’s apparently for training paratroopers and owned by a defense company to help military training. So, maybe not so funny. But it still looks ridiculous. It flew over Minneapolis back on february 8th

Route on adsb.lol

747 Dreamlifter

Skystats tags this as “Absolute Unit”, and you know what, it sure is.

The Boeing 747 dreamlifter was designed to be able to ship the Boeing 787 parts between Italy, Japan, and the US. And it appears that’s exactly what it was doing, flying from Japan, to anchorage, down to Charlston, flying over Minneapolis en route.

Route on adsb.lol

Next steps

Some next steps I want to do:

• Learn more about antenna selection. That apparently can make a huge difference in range, but I just have the stock antenna for my SDR.
• Hook into other sources - ACARS and VDL2 could be interesting to listen in on, and it’s apparently possible with an SDR. Maybe I should get a second one to complement my ADSB listening?

Resources

Build your own feeder:

• ADSB.im raspberry pi image

Watch planes online:

• ADSBExchange
• ADSB.lol

Some interesting other things to do while tracking aircraft:

• Receiving Airplane data with ACARS
• Receiving VDL mode 2

Fun plane data:

• Skystats
• plane-alert-db

I did end up transitioning to embedded firmware development, first at Medtronic for 3 years, and then another year at Dojo Five, a small local consultancy company. I’ve gone to Teardown, a great conference that I hope to go to again this year. I’ve also had some time volunteering with the local fixit clinics, fixing small household appliances and meeting some cool people there.

I have some plans for more articles around using an oscilliscope, circuit analysis for FW engineers, getting into ADSB flight tracking, Lora and meshtastic, and more.

While you wait, here’s a couple Dojo Five Blog posts I wrote while I worked there:

An Intro to Test Equipment Automation with VISA, SCPI, and the power of Python

An Overview of the quirks of NanoPB

The Basics of going from bare metal to using the Nordic SDK Build system

I recently got my hands on an old bluetooth lightbulb that was first made in 2015. But the phone app to control it is clunky and ugly, and hasn’t been updated since 2018. So if I want to still use it now, the options are to have an unmaintaned, potential security-risk app on my phone, or not use it. Or, be me, since I’ve been itching to learn more about bluetooth low energy. In which case, it becomes time to do some reverse engineering!

The Plan

Use the adafruit bluetooth LE sniffer to watch communication between the app and the lightbulb, and see if I can figure out how to turn it on/off, change the color, and set timers.

Intro to Bluetooth Low Energy

First, the bluetooth light will advertise itself using Generic Access Profile, aka GAP, until an app or what have you connects to it. I’m not going to go into that right now, because for reverse engineering, I know it connects, I want to know what it does once it’s connected. That means I need to focus on a different protocol, Generic Attribute Profile, aka GATT.

Once a bluetooth low energy thing is connected to the app that controls it, it switches to using GATT to communicate. GATT is what allows you to interact with the lightbulb (or other peripheral device). You can read data from the lightbulb (e.g. what color it is right now), or send data (e.g. setting the color, setting the timer).

For GATT there are three concepts that are the most important: profiles, services, and characteristics. Profiles are collections of services, and services have a collection of characteristics. Characteristics are basically just key/value pairs that let you read and write data to the connected device. Services collect related characteristics (e.g. color control characteristics would all be grouped in a color service). Profiles are an abstract collection of services, not on a device itself. Services and characteristics ARE on the device.

Let’s do a made-up example of those three concepts. Say I want to make a Lightbulb Profile. Profiles are conceptual collections of services. For this lighbulb profile, I want to have two different services. One service is the Light service, which will control the light color and on/off state. The other service is the timer service. That’ll control what timers a user sets (e.g. have the light turn on at 9am). The Light service will probably need to read and write the current hex color of the lightbulb, so it’ll need a characteristic with a value of the current hex color, and permissions to both read and write. It’ll also need a second characteristic, maybe just called the power characteristic, and the value is 1 or 0, depending on if it’s on or off. I’d also set that to be both read and write, so I could see if the light is turned on, and if not, set it to on.

The Timer service will also have a collection of characteristics that will let it view current timers on the device, add/delete timers, etc.

Here’s a diagram of the above example, to give you another view. Note if you connected to the lightbulb, you wouldn’t see the profile info, just the services and characteristics.

NOTE this is not how the actual lightbulb I’m using is setup, this is just an example of a way to structure a bluetooth LE device to get the concepts in your head

There’s one final bit that is used a lot: the characteristic descriptor. This is attached to a characteristic, and is basically metadata about the characteristic. So for the earlier Lightbulb profile, I can create a characteristic descriptor that is attached to the Color characteristic, and set the descriptor to have a value of “the color of the light in hex”. This helps me if I’m troubleshooting an app to talk with the lightbulb, or when I’m reverse engineering.

The conceptual model I currently have is something like this:

The little “+” on the line indicates that the box above it can have one or more. So a profile can have one or more services in it. The “*” means it can have zero or one. So the Characteristics can have zero or more characteristic descriptors. The words tabbed in are individual fields. So Characteristics have permissions and a value.

However, that’s the conceptual idea, not the implementation. In a more in-depth way, GATT is a big table. Each row has a handle to reference it, kind of like a key in a key/value pair. Then there’s a type, which is a UUID that either matches up with the 16-bit UUID Numbers Document or is custom. Then permissions (READ, WRITE, NOTIFY), and lastly, some sort of value, which depends on the type of that row. Yes, that’s pretty of confusing.

NOTE That 16-bit UUID Numbers Document is really important. It lists all bluetooth-defined UUIDs. These UUIDs are used all over the place to identify the type of row each row in the GATT table is, types of characteristics and descriptors, etc.

Remember that conceptual model? That maps to this table, I’ll walk you through making your own GATT table. Since the profiles are abstract, we can just ignore them for now. So first, we have to declare we have a service. We do this by creating a handle (in my example below, 0x001), setting the type as “Service Declaration”, permission to read only, and then the value as a UUID that says what sort of service it is. Bluetooth has a lot of declared “GATT Service Type” UUIDs in the document I linked above, you can browse them if you want.

Now that a service has been declared, lets declare some characteristics. Instead of having a single row where the “value” column is the value of the characteristic, we first have to do a Characteristic Declaration, like we did the service. So that gets its own handle, the type is “Characteristic Declaration”, it’s read only, and the value has several bits of info:

• the handle of the characteristic value (I’ll get to that next)
• the type of characteristic it is
• the permissions for this characteristic (e.g. if it’s WRITE or READ only).

The type of characteristic is one of the “GATT Characteristic and Object Type” UUIDs in the bluetooth doc linked above, OR a custom UUID that the devs decide on.

Once we have have the Characteristic declaration we can add a row talking about the Characteristic’s value. It gets a handle, type, and permission that all have to match what the Characteristic Declaration said it would have. Finally, the value field for this row is the actual value set by the peripheral. So if this characteristic lets me read the hex color the light is currently set to, the value would be some hex value like 64ff32.

And lastly, Characteristic Descriptors! These don’t get a declaration and then a value row like Characteristics, instead they’re just a single row after the Characteristic Value row. They get a handle, the type is one of the “GATT Descriptor” UUIDs from the bluetooth doc, or a custom one set by a dev, permissions (what permissions depends on the type), and lastly a value. A common descriptor is the “Characteristic User Description Descriptor”, which is UUID 0x2901. The value for this descriptor is a user-readable description for the given descriptor. So for the example characteristic above, a characteristic description descriptor would have a value of “Hex value of light” or something.

Here’s the table I described above, but in actual table-form:

Oof that’s a lot of theory and examples, let’s get back to what I wanted to do in the first place, poke around at a lightbulb.

Exploring the Lightbulb

Now that I have some idea what I’m looking at, let’s start looking at the lightbulb. There’s several apps that help troubleshoot and look at bluetooth LE connections, I used the nordic smartphone app for android. That gave me some basic lightbulb info:

Name: LEDBlue-8B109BEB 
Address: 50:33:8B:10:9B:EB
RSSI: -46
Last advertisement packet:
Raw data: 0x0201060702F0FFE5FFE0FF12094C4544426C75652D384231303942454220051210001400020A04

The Address info is important because that is how it’ll appear in wireshark! Let’s explore some of the services and characteristics the lightbulb is advertising, to give some hints for when we start sniffing.

It has several standard services that are designed by bluetooth, the Generic Access, Generic Attribute and Device Information Service. Poking around at them, though, it looks like the characteristics in those services are mostly set to default values, or not set at all. It also has the following Unknown UUID services:

0000fff0-0000-1000-8000-00805f9b34fb
0000ffe5-0000-1000-8000-00805f9b34fb
0000ffe0-0000-1000-8000-00805f9b34fb

The format of the UUIDs follow the bluetooth standard (0000XXXX-0000-1000-8000-00805f9b34fb where XXXX is a unique value), but I looked up the services on bluetooth’s 16-bit UUID Numbers Document and didn’t find anything. So presumably these are custom services designed by the company that made the lightbulb.

I poked around in the different services, and the 0xFFE5 service seems the most promising for controlling light colors. It has several characteristics with descripters that say “Green” or “Red” or “RGBW 4 bytes”. The individual colors have READ, WRITE permissions, but the RGBW characteristic is WRITE NO RESPONSE, which is interesting to note.

The other unknown services are not has helpful, and most of the characteristics have generic “Characteristic 2” descriptors attached to them. Apparently the devs got tired of giving useful descriptions at this point?

I think this is about the limit of interesting things I can do just looking at the nordic app. Let’s switch to sniffing!

Setting up the Sniffer

Once again, Adafruit has a nice writeup on the setup for the bluetooth sniffer.

The basic steps I did was:

1. downloaded extcap folder from adafruit
2. copied into my extcap folder for wireshark, which for me was at: /usr/lib/x86_64-linux-gnu/wireshark/extcap
3. ran python --version to make sure it was 2.7
4. tested by going to the extcap directory in terminal and running python nrf_sniffer.py
5. ran sudo pip2 install pyserial to get rid of serial error
6. tested again and got “No arguments given!” which indicates it’s running right

Once the initial setup was done, I closed wireshark, plugged in the sniffer, re-opened wireshark, and found the sniffer listed as “nrf-Sniffer” in the list of local interfaces on wireshark. But when I double clicked to run, I got the error: “Couldn’t run /usr/bin/dumpcap in child process: Permission denied”. Turns out my account wasn’t included in the wireshark group, so I had to update it by running sudo usermod -a -G wireshark danielle. Once I logged off and then logged on again it worked. I was sniffing packets!

NOTE Nordic semiconductor has a user guide that adafruit saved a copy of, it’s worth a read: https://cdn-learn.adafruit.com/assets/assets/000/059/041/original/nRF_Sniffer_User_Guide_v2.1.pdf?1533935335

Sniffing Packets

During the initial sniffing to see what’s up, I was really frustrated by something so I want to highlight it right away. The sniffer is not guarenteed to catch every packet on the air. So sometimes, it will miss the connection request from the phone to the lightbulb. This means it can’t track that connection, and wireshark wont display any packets after the phone is connected, even if it’s sending lots of data back and forth. If you connect to the lightbulb (or other bluetooth LE device) and wireshark stops tracking it, close the app on your phone, wait for the lightbulb to start advertising again, and try to connect again.

Following One Device

Once the wireshark session is started, you’ll probably be slammed with a bunch of advertising packets from every peripheral device that isn’t currently connected. The nordic add-on for wireshark has a drop down labeled “Device”, which lists the Mac address of all advertising addresses. Select the lightbulb 50:33:8B:10:9B:EB and it’ll only display those packets.

So now it only displays the lightbulb’s packets, but there’s still a lot to read through. Let’s start using some filters.

Parsing a capture

Like I said earlier, I don’t care about the advertising and connecting process. I can’t imagine a lightbulb would have a non-standard connection, so I’m just going to ignore it for now. Wireshark lets you filter out packets by writing statements in the “Apply a display filter” textbox at the top. To use a filter to remove advertising packets, type in !btle.advertising_address and hit enter.

This makes it a bit more readable, but still has packets with protocol “LE LL”, which stands for bluetooth Low Energy Link Layer. This is responsible for advertising, scanning, and creating/maintaining connections, and other protocols rely on it. So again, I don’t think I care about this layer. In fact, I really care about the GATT stuff, so lets only display that. The display filter for that is just btatt.

NOTE The Source column switches to “Master”/”Slave” terminology automatically. The “Master” is phone, “the slave” is the lightbulb. I’m not sure how to change that text to a more descriptive option like “Central” and “peripheral” or “client” and “server”, unfortunately.

This is a more manageable list. Poking around, we can see initially the phone is sending info requests about different services and characteristics. Then, right at the end of the capture, there’s a bunch of “sent write command” to the 0x002e handle. Changing the filter to btatt.opcode == 0x52 filters the ATT packets down to just the ones sending write commands. The info column gives me the handle it’s writing to, but not the value it’s sending. I can add the value as a column by opening up the packet in the bottom window (aka click on a write packet), right-click on the value, and select “apply as column”.

NOTE I did some cheating on the screenshots, and they’re actually from the capture when I did my testing below. So if you notice any patterns already, pretend you didn’t yet, I’ll talk more about that in the next section!

Now that I’ve poked around some, let’s do some testing.

Hypothesis and Testing

The first thing I want to figure out is something I would think is the simplest: setting the color of the lightbulb. The lightbulb app lets you set the color by selecting a point on a color wheel and set brightness with a slider bar at the bottom of the screen. The RGB of the color you select is displayed in the upper left. You can also save up to 5 specific color/brightness settings and switch to them with a touch of a button. Here’s a screenshot:

So, my hypothesis: It’s probably sending color data via either RGB or Hex. To easiser find those packets, I can save specific colors, varying the color and brightness, and then sniff the connection while repeatedly cycling through my saved colors.

Here’s the color options I saved, their RGB, Hex, and Brightness values:

NOTE: I made option 3, then set the brightness to 50% and then set that as option four. So it should have had the same RGB values. Since it didn’t, that must mean the app code changed the RGB value based on brightness. So brightness may just be a function of RGB and not a separate variable at all.

With that set, I disconnected from the lightbulb, started a wireshark session, connected, and cycled through the saved colors I just added, going through them in order 3 times.

I stopped the capture and did the filtering I talked about above. Looking at the write commands to the 0x02e handle, and just the values sent, I see a pattern:

Value 
5664ff3200f0aa
56ff653500f0aa
566633ff00f0aa
5633198000f0aa

5664ff3200f0aa
56ff653500f0aa
566633ff00f0aa
5633198000f0aa

5664ff3200f0aa
56ff653500f0aa
566633ff00f0aa
5633198000f0aa

All values start with 56 and end with 00f0aa. I cycled through the 4 options three times in a row, and I can see these values repeating three times. Well that seems awfully suspicious…

Sending Custom Commands

As it turns out, my laptop already has all the hardware and software needed to send custom bluetooth LE packets! It’s a linux box, and already has bluez on it, as well as gattool, hciconfig, and hcitool. If you also have a linux box but not the tools yet, Jared Wolf has a blog post about how to install and set them up: https://www.jaredwolff.com/get-started-with-bluetooth-low-energy/.

If you don’t have a linux box, Adafruit does have a tutorial on how to install the bluez stack on a raspberry pi here: https://learn.adafruit.com/pibeacon-ibeacon-with-a-raspberry-pi/compiling-bluez.

Basic steps for use:

1. Make sure bluetooth is turned on on computer! In my case, I go to where I set the wifi and make sure the little bluetooth symbol has the word “On” next to it.
2. In terminal, run hcitool dev, which shows any bluetooth hardware connected to my laptop, and it’s status. I took a screenshot of running this command when the bluetooth is turned off, then turned on, to see the difference:
3. Running sudo hcitool lescan does a low energy scan with the default bluetooth device (my laptop’s hci0 device) and shows the IDs of any broadcasting peripherals. I spotted the 50:33:8B:10:9B:EB device so I know my laptop is within range.

   NOTE if sudo hcitool lescan doesn’t return anything, you may have to turn off bluetooth and turn it back on again. I had that happen a couple times and it was unclear why that happened.

4. Run sudo gatttool -I to start an interactive gatttool instance, and then enter connect 50:33:8B:10:9B:EB to connect and start interactive session with lightbulb. For help with gatttool, type help

Using the gatt tool’s characteristics command I get a list of all characteristics and their UUIDs. The nordic app listed the RGBW 4bytes characteristic as a UUID of 0xFFE9. The characteristics command gives this line with that UUID:

handle: 0x002d, char properties: 0x04, char value handle: 0x002e, uuid: 0000ffe9-0000-1000-8000-00805f9b34fb

This means the characteristic value’s handle is 0x002e, and it’s the one I’d want to write to if I want to update the RGBW 4bytes characteristic. It also matches up nicely with wireshark’s packet capture. So the lightbulb is sending data to the RGBW 4byte characteristic! This further supports my “it’s sending hex to update the color” hypothesis.

If I only had the wireshark data, I could also work backwards to find out the characteristic it’s writing to. Running primary gives me a list of primary services, and the range of handles for each. Looking at that I can see the 0xffe5 service has handles 0x0023 to 0x0033, so the 0x002e handle is in that service. Relevant line:

attr handle: 0x0023, end grp handle: 0x0033 uuid: 0000ffe5-0000-1000-8000-00805f9b34fb

So I can then run characteristics 0x0023 0x0033 and see all characteristics:

handle: 0x0024, char properties: 0x0a, char value handle: 0x0025, uuid: 0000ffe6-0000-1000-8000-00805f9b34fb
handle: 0x0027, char properties: 0x0a, char value handle: 0x0028, uuid: 0000ffe7-0000-1000-8000-00805f9b34fb
handle: 0x002a, char properties: 0x0a, char value handle: 0x002b, uuid: 0000ffe8-0000-1000-8000-00805f9b34fb
handle: 0x002d, char properties: 0x04, char value handle: 0x002e, uuid: 0000ffe9-0000-1000-8000-00805f9b34fb
handle: 0x0030, char properties: 0x0a, char value handle: 0x0031, uuid: 0000ffea-0000-1000-8000-00805f9b34fb

And I can see the characteristic value handle of 0x002e is for characteristic with handle 0x002d! I can also see that the next characteristic’s handle is 0x0030, not 0x002f. Handles don’t have to be consecutive, but this may mean there’s a characteristic descriptor after the characteristic value. I can search for this by doing char-desc 0x002e 0x0030. And it gets me a result, there is a handle 0x002f, and the UUID of 00002901-0000-1000-8000-00805f9b34fb means it’s a “Characteristic User Description”. Let’s read it with char-read-hnd 0x002f, which gives us:

Characteristic value/descriptor: 52 47 42 57 20 34 20 62 79 74 65 73 

And that hex value, when dropped into a hex to ASCII converter, gives us “RGBW 4 bytes”. Yeah that’s a lot of work, I think I’ll stick with the phone app for exploration. Moving on to the fun bit!

I know I should send a write command to the 0x002e handle, and if I want to send a color, I need to have the hex value, surrounded by unknown, seemingly hard-coded values. So sending two different colors would be:

char-write-cmd 0x002e 5664ff3200f0aa
char-write-cmd 0x002e 5633198000f0aa

Hit enter on one of them, and…IT WORKS!!!!!!

After I danced around a bit, I tested it a bit more. Looks like the command to send arbitrary color, where XXXXXX is hex color, is: char-write-cmd 0x002e 56XXXXXX00f0aa. The extra bytes before and after the hex color don’t seem to matter much, though without them it doesn’t work. Something to signify to the lightbulb I’m sending a hex value, I guess.

Other Commands

Using the same process described above, I also looked into the ability to turn the lightbulb on and off, and the default “Warm White” and “Cool White” settings. They all also send to the 0x002e handle, just sending different values. My discoveries:

• Turn off Light: char-write-cmd 0x002e cc2433
• Turn on Light: char-write-cmd 0x002e cc2333
• Set to “Warm white”: char-write-cmd 0x002e 56000000ff0faa
• Set to “Cool White”: char-write-cmd 0x002e 56ffffff00f0aa

For the warm white, I can see it sets the hex color to all zeros, but sets the two zeros after the hex color to ff. This implies to me there’s actually two sets of lights in there. One does the color, and one does the “Warm white” light. Sadly I only have one lightbulb, otherwise I’d be tempted to do a teardown to see if I’m right…

Conclusion

I had my first foray into bluetooth Low Energy, and got to do some real reverse engineering! I haven’t figured out the timers yet, since it’s sending more complicated data, and I need to do more mucking about to decode what it’s sending. But this is a good place to end this post (It’s getting long!), so if I do figure it out, it’ll be a seperate post. Doing this was equal parts fun and frustrating, and I definitely feel like I understand bluetooth LE way more than I did before.

Resources

Adafruit has an intro to bluetooth page if you want a good overview: https://learn.adafruit.com/introduction-to-bluetooth-low-energy

Bluetooth’s developer guide: https://www.bluetooth.com/blog/a-developers-guide-to-bluetooth/

General bluetooth low energy info:https://www.bluetooth.com/bluetooth-resources/bluetooth-le-developer-starter-kit/

Bluetooth Low Energy in 10 minutes video: https://www.youtube.com/watch?v=JSQhRyTKnW4

Nordic semiconductors “devzone” site has some well written info about low energy too: https://devzone.nordicsemi.com/nordic/short-range-guides/b/bluetooth-low-energy/posts/ble-characteristics-a-beginners-tutorial

Getting Started with Bluetooth Low Energy from Oreilly has chapter 4 online, which is about GATT: https://www.oreilly.com/library/view/getting-started-with/9781491900550/ch04.html

Good writeup on using gatttool and hcitool to connect to bluetooth LE devices: https://www.jaredwolff.com/get-started-with-bluetooth-low-energy/#connect-to-your-bluetooth-low-energy-device

Bluetooth has a list of 16-bit UUID Numbers and what they map to. It used to be all on searchable webpages, but they’ve moved it to a PDF: https://btprodspecificationrefs.blob.core.windows.net/assigned-values/16-bit%20UUID%20Numbers%20Document.pdf

The Plan

Do some basic interrupts and learn more about how to use them with the ESP8266. A basic interrupt example that I’ve seen around a lot is turning on an LED for a certain amount of time when a button is pressed. So let’s do that!

NOTE I’m assuming in this post you already have the setup to run the Arduino IDE and write programs for the ESP8266.

Hardware

Hardware needed:

• Adafruit Feather Huzzah w/ ESP8266, with soldered on headers
• Breadboard
• Jumper wires
• LED
• 300 Ohm Resistor

To wire up the circuit I’ll be using, attach the LED and resistor in series, and connect to pin 14 and ground on the feather. Then attach the momentary switch to pin 4 and ground. Below is a simple drawing showing the circuit. The wire color in the diagram is just for visual clarity.

Without Interrupts

First, lets get the basic functionality working without interrupts. Note that I set the momentary switch as a pull up. This means when the switch is not pressed, I’ll read the pin as HIGH, and when it’s pressed, it’ll go to LOW. If I didn’t set it, the pin would not be guaranteed either HIGH or LOW, and reading the state of pin 4 wouldn’t tell me if the button was pushed or not!

I also used the millis() function instead of delay(). This is because delay doesn’t let me do anything else while I’m waiting for the LED to finish being on. If I want to use this code to do other things while the LED is on, I have to instead track the time since it was turned on, so I can do other things in the loop() too.

#define GPIO_INTERRUPT_PIN 4
#define LED_PIN 14
#define WAIT_TIME 500

unsigned long lastTrigger = millis();
bool buttonPressed = false;

void setup() {
  //Start serial
  Serial.begin(115200);
  Serial.println("Starting sketch");

  //Set the LED pin to output
  pinMode(LED_PIN, OUTPUT);
  //set the eventual interrupt pin as an input that is pulled up
  pinMode(GPIO_INTERRUPT_PIN, INPUT_PULLUP);

}

void loop() {

  if (!buttonPressed && digitalRead(GPIO_INTERRUPT_PIN) == LOW){
    Serial.println("Button pressed");
    digitalWrite(LED_PIN, HIGH);
    buttonPressed = true;
    lastTrigger = millis();
  }

  if (buttonPressed && lastTrigger + WAIT_TIME < millis()){
    digitalWrite(LED_PIN, LOW);
    buttonPressed = false;
  }

  //Other functionality goes here...

}

Of course most code isn’t only checking a button. Let’s add a delay in there to simulate doing other work.

#define GPIO_INTERRUPT_PIN 4
#define LED_PIN 14
#define WAIT_TIME 500

unsigned long lastTrigger = millis();
bool buttonPressed = false;

void setup() {
  //Start serial
  Serial.begin(115200);
  Serial.println("Starting sketch");

  //Set the LED pin to output
  pinMode(LED_PIN, OUTPUT);
  //set the eventual interrupt pin as an input that is pulled up
  pinMode(GPIO_INTERRUPT_PIN, INPUT_PULLUP);

}

void loop() {

  if (!buttonPressed && digitalRead(GPIO_INTERRUPT_PIN) == LOW){
    Serial.println("Button pressed");
    digitalWrite(LED_PIN, HIGH);
    buttonPressed = true;
    lastTrigger = millis();
  }

  if (buttonPressed && lastTrigger + WAIT_TIME < millis()){
    digitalWrite(LED_PIN, LOW);
    buttonPressed = false;
  }

  //Other functionality goes here...
  delay(500);

}

If you upload this code to the feather, you can see the button press is erratic. First, if you mess with the delay time, you can see that once the number of milliseconds you delay is longer than WAIT_TIME, the LED will stay on longer. This is because turning off the LED is in the loop() function with everything else, so while it’s in the delay, the code can’t turn the LED off!

Second, it doesn’t always register that the button was pressed. When it’s in the delay function (aka doing other stuff), it’s not checking if the button has switched to LOW (aka been pressed). Completely missing a button press is not acceptable for pretty much any project! Let’s fix that.

Basics of Interrupts

Interrupts are a way to interrupt the standard flow of your program. In my particular case, I want to interrupt whatever I’m doing in my loop() function and turn on the LED when I press the button.

The steps of a generic interrupt happening are:

1. The ESP8266 is running code (both overhead and code in my loop function)
2. Something happens that I told the ESP8266 in my setup function needs to trigger an interrupt
3. The ESP8266 stops whatever is currently happening and saves the context so it can get back to it later
4. The ESP8266 looks up what it’s supposed to do when the trigger happens (aka looks for my interrupt handler code)
5. It runs my handler code
6. Once that’s done, it pulls the saved context back into its proper place, and starts up whatever it was doing before the interrupt happened.

I also made a timing diagram of the button press interrupt happening, to give a more visual represenation.

So, to make this interrupt I want a reality, I have some basic setup to do:

1. Define what I want to happen when the interrupt triggers (i.e. turn on the LED and set some variables to control how long the LED is on). This will be my interrupt handler function, also known as an interrupt service routine or ISR.
2. Set how I want to trigger the interrupt. In my case, I want to trigger on a GPIO pin.
3. Since I’m triggering off of a GPIO pin, I need to set what state I want to trigger on. Some example options would be rising (when it goes from LOW to HIGH), falling (when it goes from HIGH to LOW), or whenever there’s a change.

Arduino Specifics

Arduino has a builtin function attachInterrupt that lets me set what pin and when to trigger the interrupt. They have a write up here: https://www.arduino.cc/reference/en/language/functions/external-interrupts/attachinterrupt/. It’s pretty straightforward. Call attachInterrupt with the pin to watch, the function to call when it triggers, and when to trigger it.

Arduino has default constants you can use for when to trigger the interrupts. They are:

LOW to trigger the interrupt whenever the pin is low,

CHANGE to trigger the interrupt whenever the pin changes value

RISING to trigger when the pin goes from low to high,

FALLING for when the pin goes from high to low.

Since I want to trigger off my button attached to pin 4, and I want to trigger it when the pin goes from LOW to HIGH (aka when a person lifts their finger off the button), I can setup the interrupt with the below code:

attachInterrupt(digitalPinToInterrupt(GPIO_INTERRUPT_PIN), detectsButton, RISING);

That detectsButton argument is the name of the interrupt handler function. Which has to be defined a little different from what the arduino documentation says because I’m using the ESP8266.

ESP8266 Specifics

The ESP8266 has a slightly different setup than the standard chips used in Arduinos, which results in an extra step. When declaring the interrupt function, you have to prepend ICACHE_RAM_ATTR to the function definition. This tells the linker to put this function in RAM instead of flash, where the rest of the program goes. The exact reasons why seem a bit murky. Different posts and forum questions online give no reason beyond “you gotta put this here”, or give different reasons why. But the basics as I understand it is that the interrupt handler (aka the code that runs during the interrupt) needs to be stored in RAM on the ESP8266, not flash, where it would normally be stored without that ICACHE_RAM_ATTR. This is because the interrupt code can run at any time, and if the chip is in the middle of writing or reading to flash, trying to also run the interrupt handler code in flash at the same time will cause the chip to crash. Crashing is bad, so the ESP8266 Arduino code has a built-in check to make sure all interrupt handlers are in the RAM. If it’s not, the code will pre-emptively crash with an error that says “ISR not in IRAM!”.

So the detectsButton function is defined as follows:

ICACHE_RAM_ATTR void detectsButton() {
  //NOTE because in ISR, millis() call will return the same value every time
  
  //only trigger if I didn't just trigger DEBOUNCE_TIME ago
  //lastTrigger > millis() means millis has overflowed
  if (lastTrigger > millis() || lastTrigger+DEBOUNCE_TIME < millis())
  {
    lastTrigger = millis(); 
    buttonPressed = true;
  }
  else{
    bounce++;
  }
 
}

There’s a couple things to note here. One, my comment that starts with “NOTE” that talks about the millis function. The millis() function returns the number of milliseconds it’s been since the chip turned on. It actually uses interrupts in order to update! That interrupt is setup by the Arduino code, which is why I don’t have to setup that interrupt in my code. However, Arduino disables interrupts when you’re running an interrupt handler. Presumably so you don’t get interrupts triggering while you’re handling an interrupt and just get lost in nested interrupts forever. But this means the interrupt to update the return value for millis() wont run, and so it’ll return the same value the entire time you’re in an interrupt handler.

Second, there’s a check lastTrigger+DEBOUNCE_TIME < millis() that happens before it actually updates that the button was pressed. The ESP8266 says that the button has changed from LOW to HIGH, which is why it’s running this code, so why am I adding extra checks? It’s because of debouncing. The analog world can be messy (citation needed), so when the button is first pressed, or first released, the reading on pin 4 can bounce between LOW and HIGH a bit before it settles down on one or the other. During that time, the interrupt can get triggered several times. But I only want it to actually trigger once! So I add a check in there. If the last time the button was triggered was really recent (aka within DEBOUNCE_TIME milliseconds), then I assume this trigger was the button bouncing a bit, and ignore it. Because I was curious, though, I don’t completely ignore it. Instead I set the bounce++ to add to my total bounce amount. In my loop code I check that variable and output to serial that I bounced, just so I could see how often it happens.

Lastly, there’s a check for lastTrigger > millis() which at first glance doesn’t make sense. ThelastTrigger variable is set to millis() from the last time it was triggered, and the number of milliseconds since the chip was turned on will only go up (we’re not time-traveling!). So how can this check ever return true? The answer is overflow! The number of milliseconds since turn on is stored in an unsigned long variable. Since it’s unsigned, the variable is always positive, and since it’s a long, its size is 32 bits. This means the max number of milliseconds it can store is 2³² - 1, aka 4294967295 milliseconds. Convert that to days, and you get 49.7103009259 days, aka around 50 days (which the Arduino documentation on millis() says). Once it gets to that number, and you add one more millisecond, the number is 33 bits long! So the code just gets rid of the topmost number, and suddenly you’re getting 0 milliseconds from millis(). Now, I’m unlikely to run this test code for 50 days, so I don’t really need it, but I like to be complete when I can.

Final Code

Now that I have all that preamble out of the way, I can put together final example code using interrupts on the ESP8266.

#define GPIO_INTERRUPT_PIN 4
#define LED_PIN 14
#define WAIT_TIME 250
#define DEBOUNCE_TIME 10


volatile bool buttonPressed = false;
volatile unsigned long lastTrigger = millis();
volatile int bounce = 0;
int prevBounceCount = 0;

ICACHE_RAM_ATTR void detectsButton() {
  //NOTE because in ISR, millis() call will return the same value every time
  
  //only trigger if I didn't just trigger DEBOUNCE_TIME ago
  //lastTrigger > millis() means millis has overflowed
  if (lastTrigger > millis() || lastTrigger+DEBOUNCE_TIME < millis())
  {
    lastTrigger = millis(); 
    buttonPressed = true;
  }
  else{
    bounce++;
  }
 
}

void setup() {
  Serial.begin(115200);
  Serial.println("Starting sketch");
  pinMode(LED_PIN, OUTPUT);
  pinMode(GPIO_INTERRUPT_PIN, INPUT_PULLUP);

  attachInterrupt(digitalPinToInterrupt(GPIO_INTERRUPT_PIN), detectsButton, RISING);

}

void loop() {

  //lastTrigger > millis() means millis has overflowed
  if (buttonPressed && (lastTrigger > millis() || lastTrigger+WAIT_TIME <= millis())){
     Serial.println("button was pressed, time to turn off LED");
     digitalWrite(LED_PIN, LOW);
     buttonPressed = false;
  }
  else if (buttonPressed){
    digitalWrite(LED_PIN, HIGH);
  }
  if (bounce != prevBounceCount){
    Serial.println("Debounced");
    prevBounceCount = bounce;
  }
  
  //Other functionality goes here...
  delay(250);

}

Notice how several variables are declared to be volatile. This is important! Setting variables to volatile is me telling the compiler that the variables may change at any time outside the standard code flow. I set them to volatile in the code so the compiler knows not to optimize them out, or to used a cached value. In this case, I’m setting several variables to volatile because I set them in the interrupt handler. Which means every time I access them in the loop() function, it’s possible that the interrupt had happened just before it, and changed those variables! This is important info for the compiler to know, and makes it so the compiled version of my code is actually what I want it to be. Embedded FM has a short article about volatile if you want to learn more: https://embedded.fm/blog/2017/2/23/explaining-the-c-keyword-volatile

Now when a button is pressed (and released), the LED turns on immediately! Take some time to repeatedly press the button to turn on the LED and marvel at your new-found knowledge. Note, however, that this works completely as intended only as long as the other functionality in the loop() doesn’t take longer than WAIT_TIME to complete. If the other functionality takes longer, than the if statement checking how long it’s been since last trigger won’t run in time, and the LED will stay on longer than WAIT_TIME. This is because turning the LED off is in the loop function, and so still restricted by all the other work done there. If you want an exact timing for how long the LED is on, then you’d have to add an interrupt that triggers on time, and that is outside the scope of this post. The ESP8266 does have timer interrupts, but that requires an extra library and more testing/playing around, so I’ll leave that as a potential future post.

In the meantime, enjoy your new knowledge, and go forth and interrupt things!

References

• Arduino’s way of adding interrupts: https://www.arduino.cc/reference/en/language/functions/external-interrupts/attachinterrupt/
• In-depth article about Arduino interrupts (focused on AVR but applicable to ESP8266): http://gammon.com.au/interrupts
• ESP8266 and Arduino specific interrupts: https://randomnerdtutorials.com/interrupts-timers-esp8266-arduino-ide-nodemcu/
• Arduino Documentation for millis(): https://www.arduino.cc/reference/en/language/functions/time/millis/
• ESP8266 Arduino interrupt documentation: https://arduino-esp8266.readthedocs.io/en/latest/reference.html#interrupts
• The book Making Embedded Systems by Elicia White has a section talking about interrupts that I referenced while writing this post: https://www.oreilly.com/library/view/making-embedded-systems/9781449308889/

The Hardware

I bought the rotary encoder + extras from adafruit. I had originally thought to grab one from digikey or elsewhere (if only to get used to buying from other places), but the sheer amount of variables I had to know about frazzled me. How many detents are enough? What are pulses per revolution? Adafruit just gives you one option. It’s a rotary encoder, what more do you need to know! Adafruit provides a one and done, and sometimes, that’s what you want to do. Once I got it, I put the knob on it and gave it a twirl. A very satisfying clicky twisting motion, exactly what I wanted! Now to do some reading.

Adafruit provides the datasheet, so I can see what’s what.

The 3 connector prongs on one side are the A, B and common (aka ground) channels. If I connect up to them and monitor A and B as I turn the knob, I should see the pattern specified in the Quadrature table.

Since I started only kind of understanding this information (and frankly was just confused by that signal schematic), I decided to do some testing. Since I wanted to use the encoder with the adafruit feather huzzah with ESP8266, I wired up the simplest circuit I could, connecting channel A on the encoder to pin 4, channel B to pin 5, and the common to ground. Diagram below:

The First Test

The simplest test is to see what happens to things on channel A and B when I turn the knob. If you look back at the Quadrature table, you can see when both signals are HIGH, it means it’s off. And the way I wired it requires the pins to be pullups. So for this rotary encoder, when nothing is happening, they’re both set to HIGH, and when I turn the knob, they go down to low, one right after the other.

#define PIN_ENCODER_A      4
#define PIN_ENCODER_B      5

void setup() {
  Serial.begin(115200);
  pinMode(PIN_ENCODER_A, INPUT_PULLUP);
  pinMode(PIN_ENCODER_B, INPUT_PULLUP);

}

void loop() {
  int valA = digitalRead(PIN_ENCODER_A);
  int valB = digitalRead(PIN_ENCODER_B);

  //since these are default HIGH (aka have a pullup resistor)
  //either being set to LOW means something is happening on the encoder
  if (valA == LOW || valB == LOW){
    Serial.print(valA);
    Serial.print(" ");
    Serial.println(valB);
  }

}

With this code running on my feather, I can turn the know clockwise and counter clockwise, and see what I get.

Turned clockwise, for each “click” of the knob, I get this:

If I want, I can just put those two values next to each other and get a binary number. So it goes from 11, to 01, to 00, to 10, to 11. With the decimal equivalent being: 3, 1, 0, 2, 3

So if I track the previous state and the current state, I can tell it’s going clockwise if any of these are true:

previous state->current state
3->1
1->0
0->2
2->3

From this I can see I’ll have to at least track current and previous state, because it matters what state it’s coming from. Just knowing current state is “0” doesn’t mean anything about direction.

Let’s see if that pattern is different when it’s turned counter-clockwise. The values end up being:

Which is a different pattern! That’s good, otherwise I’d have to come up with a completely different way of figuring this out. In decimal, it would be: 3, 2, 0, 1, 3

So it’s going counterclockwise if:

3->2
2->0
0->1
1->3

Second Test - Lookup tables

Since I’ll have to do this in code eventually anyway, let’s do 1 to mean clockwise and -1 to mean counter clockwise. I can then list out all possible combinations of a previous and current state, and determine which way I’m going, or if I’m missing anything.

Based on this, I can see there’s a lot of options that don’t mean the encoder went clockwise OR counter clockwise. Looking at those undefined values, there’s two separate groups. If the previous and current values from the encoder are the same, that just means I read the values so fast, it didn’t have time to change (or in the case of 3, the encoder isn’t moving at all). So those just mean it didn’t move. For the other group of undefined values, they’re harder to categorize. Going from 0 to 3 means EITHER I moved clockwise so fast I went from 0 to 2 to 3 before my code had a chance to read the encoder values, OR I moved it counter clockwise so fast I went from 0 to 1 to 3. So that tells me the encoder rotated, but I don’t know which way! I’m not sure how to solve that, so I’m just going to treat that as a different “SKIPPED VALUE” category.

I can reorganize the data to make it a bit more readable (at least to me), by putting the previous value as the first column, and the current value as the first row, like this:

Well hey, that looks an awful lot like a lookup table, a 4x4 array with the indices corresponding to the values. Let’s do that! For the “DIDN’T MOVE” category, I can just set it to 0, and for when I skip values, I’ll do 2, a clearly wrong value. With that, I get the below table:

{0, -1, 1, 2}
{1, 0, 2, -1}
{-1, 2, 0, 1}
{2, 1, -1, 0}

With this planning, I can now try some code out and see if it works.

#define PIN_ENCODER_A      4
#define PIN_ENCODER_B      5

int prevVal = 0;
int newVal;

int lookupTable[4][4] = { {0, -1, 1, 2},
                          {1, 0, 2, -1},
                          {-1, 2, 0, 1},
                          {2, 1, -1, 0} };

void setup() {
  Serial.begin(115200);
  pinMode(PIN_ENCODER_A, INPUT_PULLUP);
  pinMode(PIN_ENCODER_B, INPUT_PULLUP);

  int valA = digitalRead(PIN_ENCODER_A);
  int valB = digitalRead(PIN_ENCODER_B);
  prevVal = (valA << 1) + valB;

}

void loop() {
  int valA = digitalRead(PIN_ENCODER_A);
  int valB = digitalRead(PIN_ENCODER_B);
  newVal = (valA << 1) + valB;

  int info = lookupTable[prevVal][newVal];


  if (info == 1){
    Serial.print("clockwise ");
    Serial.print(prevVal);
    Serial.print("->");
    Serial.println(newVal);
  }
  else if (info == -1){
    Serial.print("counter clockwise ");
    Serial.print(prevVal);
    Serial.print("->");
    Serial.println(newVal);
  }
  else if (info == 2){
    Serial.println("skipped a value");
  }
  
  prevVal = newVal;
}

Final Code - Tracking State

This correctly understands clockwise and counterclockwise movement. HOWEVER, there’s two issues. One, it’s triggering roughly 4 times each detent, and Two sometimes the code manages to get a bounce or something else wonky, and it reads, say, 3 clockwise and one counterclockwise movement while I’m moving it clockwise. So I want to track all the states it passes through, and then figure out which way its going, ignoring those occasional misreads.

NOTE “detent” is a fancy word for that “click” feeling you get with rotary encoders. Each “click” position is a detent.

#define PIN_ENCODER_A      4
#define PIN_ENCODER_B      5

int prevVal = 0;
int newVal;
unsigned int clockState = 0;
unsigned int counterClockState = 0;

//lookup table, first index is previous value
//second index is current value
//says if it's part of the sequence when moving
//clockwise (1) or counterclockwise (-1)
//didn't move (0) or skipped a value (2)
int lookupTable[4][4] = { {0, -1, 1, 2},
                          {1, 0, 2, -1},
                          {-1, 2, 0, 1},
                          {2, 1, -1, 0} };

void setup() {
  Serial.begin(115200);
  pinMode(PIN_ENCODER_A, INPUT_PULLUP);
  pinMode(PIN_ENCODER_B, INPUT_PULLUP);

  int valA = digitalRead(PIN_ENCODER_A);
  int valB = digitalRead(PIN_ENCODER_B);
  prevVal = (valA << 1) + valB;

}

void loop() {
  int valA = digitalRead(PIN_ENCODER_A);
  int valB = digitalRead(PIN_ENCODER_B);
  newVal = (valA << 1) + valB;
  
  int info = lookupTable[prevVal][newVal];


  if (info == 1){
    clockState |= (1 << newVal); //set the bit to 1
  }
  else if (info == -1){
    counterClockState |= (1 << newVal);
  }
  else if (info == 2){
    Serial.println("skipped a value");
  }

  if (prevVal != newVal && newVal == 3){
    //changed to the non moving state, lets figure out what direction we went!

    //for each clockwise and counterclockwise, the encoder state goes through 4 distinct states
    //make sure it's gone through at least 3 of those (and assume if one is missing it's because I didn't read fast enough)
    if (clockState == 0b1011 || clockState == 0b1101 || clockState == 0b1110 || clockState == 0b1111){
      Serial.println("Result was clockwise");
    }
    if (counterClockState == 0b1011 || counterClockState == 0b1101 || counterClockState == 0b1110 || counterClockState == 0b1111){
      Serial.println("Result was COUNTER clockwise");
    }
    
    clockState = 0;
    counterClockState = 0;
    
  }
  
  prevVal = newVal;
}

This seems to work! I get a single “result was blah” for every turn, and it seems to be the correct rotation, too! The remaining issue depends on the other code I use this with. This code assumes I’ll be able to read at least 3 out of the 4 states the rotary encoder goes through every time it turns left or right. If I’m checking the encoder in a main loop, and my other code goes really slow, I may only get one or two states. An alternative to this would be setting up an interrupt, so it jumps to reading the encoder every time it changes state. However, interrupts are outside the scope of this blog post, so I’ll leave that as an exercise for the reader.

Conclusion

Hopefully this gives more info on what to expect when using a rotary encoder in your future projects. It’s a lot more complicated than a potentiometer, but that lovely clicky feeling can be perfect for switching between a bunch of options. And now you have the knowledge to use them. So go forth and get clicky!

Resources

• Datasheet for the rotary encoder https://cdn-shop.adafruit.com/datasheets/pec11.pdf
• Sparkfun document talking about reading rotary encodershttps://cdn.sparkfun.com/datasheets/Robotics/How%20to%20use%20a%20quadrature%20encoder.pdf
• Learning about the Arduino IDE serial plotter https://diyrobocars.com/2020/05/04/arduino-serial-plotter-the-missing-manual/
• Adafruit’s example using a rotary encoderhttps://learn.adafruit.com/pro-trinket-rotary-encoder/example-rotary-encoder-volume-control
• Document about rotary encoders https://web.archive.org/web/20120208215116/http://www.circuitsathome.com/mcu/reading-rotary-encoder-on-arduino
• Pinout for the Adafruit Huzzah with ESP8266 https://cdn-learn.adafruit.com/assets/assets/000/046/211/original/Huzzah_ESP8266_Pinout_v1.2.pdf?1504807178

Because I’m all about learning, and also about doing things the hard way, I decided to use a different toolchain setup than what Adafruit suggests (which is just using the arduino ecosystem). Instead, I’ll go for something a little more complicated…

Plan

The plan is to use the ESP8266, in this case on the adafruit feather Huzzah with ESP8266, with the Arduino libraries, but using VSCode instead of the Arduino IDE. In the spirit of KISS (Keep It Simple Stupid), I decided to use Make to glue everything together. Happily I didn’t have to do this all from scratch, and mainly had to tweak things instead. Let’s get into it.

Benefits

Why am I even doing this in the first place? Well:

• Arduino Libraries are easy to use - I could work on getting espressif’s SDK for the ESP8266 going, and figure out how to flash the code onto the huzzah myself. But frankly, that’s a bit too much reinventing the wheel right now. I want to get something functional, and the code I have already uses Arduino libraries and calls, so why re-write it all? Plus, getting more libraries is really easy with the Arduino IDE.
• More Useful IDE - this is very subjective, but frankly, I like VSCode a lot better than the Arduino IDE. I got a file explorer, a terminal, and a text editor with syntax highlighting, what more could a gal need? Its level of flexibility means I can do everything I want in it (including writing this blog post). I don’t have to relearn a new IDE for embedded stuff, I can stick with what I know. Given all there is to know about embedded systems, I’m willing to skip the less interesting bits.
• Testing - I did googling but I couldn’t find anything about doing unit tests in the Arduino IDE. I want to build more complex programs, which means testing. Debugging boneheaded errors because I couldn’t do unit tests seemed very silly. If I’m going to debug boneheaded errors, it’s because I decided not to do unit tests. And again, there’s a lot for me to learn here, doing unit tests as I go will help me on my embedded systems adventure.
• Flexibility - If I want to have multiple builds, or add extra things to my build process, I can! I have to argue with make about it, but it’s at least possible, and kind of already a given with the tools I’m using.
• Local - I know there’s a couple online embedded IDE’s out there (Platformio comes to mind), but call me old-fashioned: I like to have code on my computer. Sure, I’ll back it up (normally to github), but having a local copy, and not relying on an online cloud service, feels like one less thing to worry about. If my code breaks or disappears, it’s because of me, not because a cloud service decided they wanted more money and paywalled me out of my code.

The Software

Arduino IDE

The arduino IDE is actually still needed for this setup. I know, I know, it seems silly to download something you don’t plan on using. But it does provide easy access to arduino libraries, plus a nice serial monitor and plotter. I don’t like coding in it, but those are nice features to have, so it’s simpler to just download it and use the bits you care about. Download the software directly from Arduino on their software download page. Make sure to select the correct version, and don’t get distracted by the web editor they’re advertising. You want the downloadable version.

VSCode

VSCode, aka Visual Studio Code, is made by Microsoft, and is completely different than their large and somewhat unwieldy Visual Studio IDE. Yes, Microsoft is still terrible at naming things. VSCode is actually open source, and free for private and commercial use. It also has a lot of plugins, built-in terminal, syntax highlighting, and is pretty lightweight for how useful it is. Think of it as somewhere between Sublime and Eclipse/Visual Studio IDE. It also runs on Mac, Linux, and Windows. You can download it from Microsoft’s visual studio code website

Makefile

Make is old-school (sorry if you’re reading this and remember when it was new. But like, it’s true. You’re old-school now. It came out in ‘76!) It’s a build automation tool that runs on the command line. You can make custom production, dev, test, whatever-else build processes. You can probably get it to clean your kitchen. The commands can be arcane and confusing, but it can do a little bit of everything if you learn the right incantation. If you have linux, you probably already have it.

The others

I’m also using git, python, and g++. The pre-made makefile also calls some perl scripts. Just to give you a full understanding of what I’m using.

The Set-up

First, make sure you have all of the above listed software. Make sure they all seem to work. Now lets start tweaking things.

Arduino Libraries

Since you want to work on an ESP8266, first you’ll have to download some libraries. The bare-minimum is the ESP8266 board library. This is from https://github.com/esp8266/Arduino, and the install instructions are pretty straightforward. Add a new board manager URL, then download the ESP8266 board.

To make sure it works, I’d suggest trying to flash one of the example sketches to the esp8266, make it blink or run a wifi scan.

VSCode

I Installed a Microsoft-written plugin that helps with syntax: https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools. The webpage shows the install instructions. Later there’s some VSCode config files you can add to your project to make its error alerts more useful.

Makefile

There’s actually two makefiles for this. The first is a pre-built one that came from https://github.com/plerup/makeEspArduino. This has install instructions, and works! But it didn’t support how I setup my project (see below), so I forked it and modified it. It’s now available on my github repo with the change I had to make.

That’s the first makefile, the second is a custom one that will live in your project. I’ll explain more below.

Project setup

Folder setup described below. -> means it’s a directory, tabs indicate which files are in what folder (e.g. “unity” folder is inside the “test” folder).

->Project name
    ->.vscode
    ->build
    ->doc
    ->test
        ->unity
            unity_internals.h
            unity.c
            unity.h
    ->src
    ->inc
    ->makeEspArduino
    makefile
    readme.md
    .gitignore
    .gitmodules

I’ll go through each of these folder and files to explain what’s in them.

makeEspArduino

This is actually a git submodule. This folder contains the https://github.com/dthurow/makeEspArduino repo. It lets you keep all your code in one place. The .gitmodules file (explained below) automatically pulls that in when you do git clone --recurse-submodules on any project you make following this setup.

.gitmodules

This lets you have git repos in folders inside other git repos. In this case, we’re adding the makeEspArduino into the project. The contents of this file are:

[submodule "makeEspArduino"]
	path = makeEspArduino
	url = git@github.com:Dthurow/makeEspArduino.git

makefile

This file does two things:

1. Sets the configuration variables that the makeEspArduino uses
2. Sets up the ability to run tests (I use the unity framework)

To make this work for you, you’ll have to make sure that:

• the LIBS directories are where your arduino libraries are.
• The SKETCH variable is set to your main file (that contains your setup() and loop() functions)
• Have g++ downloaded
• Set the BOARD to the correct value. To figure out what it should be, you can run make list_boards and the makeESPArduino file will spit out a list of accepted boards. It also has a generic option if yours isn’t on the list

DIR := ${CURDIR}
SKETCH = $(DIR)/src/[YOUR MAIN FILE HERE]
BUILD_DIR = $(DIR)/build
BOARD = huzzah
LIBS = $(HOME)/.arduino15/packages/esp8266/hardware/esp8266/2.7.4/libraries $(HOME)/Arduino/libraries 
EXCLUDE_DIRS = test
include $(DIR)/makeEspArduino/makeEspArduino.mk

#everything below here is for setting up tests
TARGET_EXTENSION=.out
#Path Definitions
PATHU = test/unity/
PATHS = src/
PATHT = test/
PATHI = inc/
PATHB = build/

#determine our source files
SRCU = $(PATHU)unity.c
SRCS = $(wildcard $(PATHS)*.cpp)
SRCT = $(wildcard $(PATHT)*.cpp)
SRC = $(SRCU) $(SRCS) $(SRCT)

#Files We Are To Work With
OBJU = $(patsubst $(PATHU)%.c,$(PATHB)%.o,$(SRCU))
OBJS = $(patsubst $(PATHS)%.cpp,$(PATHB)%.o,$(SRCS))
OBJT = $(patsubst $(PATHT)%.cpp,$(PATHB)%.o,$(SRCT))
OBJ = $(OBJU) $(OBJS) $(OBJT)

#Other files we care about
DEP = $(PATHU)unity.h $(PATHU)unity_internals.h
TGT = $(PATHB)test$(TARGET_EXTENSION)

#Tool Definitions
CC=g++
CFLAGS=-I. -I$(PATHU) -I$(PATHI) -I$(PATHS) -DTEST

test: $(PATHB) $(TGT)
	echo "running tests"
	./$(TGT)

$(PATHB)%.o:: $(PATHS)%.cpp $(DEP)
	echo "source compiling"
	$(CC) -c $(CFLAGS) $< -o $@

$(PATHB)%.o:: $(PATHT)%.cpp $(DEP)
	echo "tests compiling"
	$(CC) -c $(CFLAGS) $< -o $@

$(PATHB)%.o:: $(PATHU)%.c $(DEP)
	echo "unity compiling"
	$(CC) -c $(CFLAGS) $< -o $@

$(TGT): $(OBJ)
	echo "linking"
	$(CC) -o $@ $^

.PHONY: test

test

This folder is where you’ll put your test files (e.g. testMyCoolCode.cpp). Your tests can use the unity framework, which you can find out more about here: http://www.throwtheswitch.org/unity. You only need the three files I list in the folder setup, and you can get the latest from github https://github.com/ThrowTheSwitch/Unity/tree/master/src.

.vscode

This folder stores VSCode-specific config files that override your default configs on a project-specific basis. The makeEspArduino makefile actually has a command that auto-creates this info for you. Once the makefile and your main source file are created, you can run make vscode and it will autogenerate your .vscode folder. In my case, though, to get VSCode’s syntax checker to fully work, I had to update the includePath in the c_cpp_properties.json file. I updated mine to:

            "includePath": [
                "${env:HOME}/.arduino15/packages/esp8266/hardware/esp8266/2.7.4/tools/sdk/include",
                "${env:HOME}/.arduino15/packages/esp8266/hardware/esp8266/2.7.4/tools/sdk/lwip2/include",
                "${env:HOME}/.arduino15/packages/esp8266/hardware/esp8266/2.7.4/tools/sdk/libc/xtensa-lx106-elf/include",
                "${env:HOME}/.arduino15/packages/esp8266/hardware/esp8266/2.7.4/cores/esp8266",
                "${env:HOME}/.arduino15/packages/esp8266/hardware/esp8266/2.7.4/variants/generic",
                "${env:HOME}/.arduino15/packages/esp8266/hardware/esp8266/2.7.4/libraries/**",
                "${env:HOME}/Arduino/libraries/**",
                "${workspaceFolder}/build",
                "${workspaceFolder}/src"
                
            ],

And then VSCode no longer errored out on not seeing the Arduino Library folders.

.gitignore

This one’s simple, just ignore the build folder and the special vscode folder:

build
.vscode

src

Where your actual production code lives! I know, there’s a lot of other stuff this project has, but honest, it’s all there for a good reason.

inc

Honestly I’m on the fence if you really need this folder. This can be used to include needed header files. You can also just keep them in src, I won’t tell.

readme.md

A good file for any project, so you can write up setup or contribution info.

build

This is where the makefiles will build out any intermediary files and where it’ll put the file compiled version of the code. You can clear this folder out by typing make clean. Sometimes the default make all errors because of linker issues. In that case, run a make clean then make all.

doc

A place to put any documentation you may have! You do keep written documentation, right? Because “self-documenting code” isn’t actually a thing? Yeah of course you do, good job you.

The Final Result

Hopefully, a working project! I’ve put together the template on github https://github.com/Dthurow/generic-esp8266-project-with-vscode-and-arduino that you can pull down if you want to do that instead of building it out manually. Please do pull requests or open issues with any updates you may have, I’m curious if this project will be useful for others, or if there’s other ways of doing things I just don’t know about.

Good luck and happy programming!