Multicast DNS Tribulations

For my m8rScript project I really want to use Bonjour. It would give me a way of finding many ESP8266 devices on the network without having to nail down IP addresses. The concept is simple: you give your device a name, like yardweather. Then you type yardweather.local into your web browser and up pops the web page for your device. Additionally, you can broadcast a message on your network asking if any device supports a given service, and any device that does sends back information telling you about itself. So you can know how you are allowed to talk to it. Simple, right?

NO

I tried lots of options for the ESP8266. BUT NONE OF THEM WORKED! To make matters worse The documentation is terrible. In fact, maybe the reason none of the implementations work is because of the bad documentation.

Ok, I’m being a little unfair. All of the implementations work somewhat, but none have all the features to let you achieve the Nirvana of being able to both find out how to contact the device and what the device is capable of. So I started on a trek, first to understand what mDNS is all about and second to write an implementation for the ESP8266.

What is mDNS?

It goes by many names: Multicast DNS is it’s official IETF name, as documented here. Apple invented it and they call it Bonjour. They originally wanted to call it Rendezvous, but that name was taken, so they switched to the equally French Bonjour. And all this falls under the umbrella of Zero Configuration Networking (zeroconf).

When I started looking around I would search for “mdns packet format” and things like that and it would keep coming back with DNS hits. It didn’t occur to me at the time that mDNS IS actually a particular flavor of the Domain Name System (DNS), which is pretty much the backbone of the web. With that new understanding, a lot of the terminology and descriptions made a lot more sense. You ask a question (“what is the IP address for this domain name“) and get back an answer (“1.2.3.4”). Then you can contact that IP address and start a conversation. The two interesting bits about mDNS are a) the types of questions you ask and the responses you get back, and b) the fact that it uses multicast. Normally you ask DNS questions of a server with a particular IP address, which is added to your computer when it is first set up. Multicast just sends its questions out into the universe and then waits a while for answers to come back, often from many different sources.

For the ESP8266 the multicast part is easy. It uses a networking technology called UDP, which is well supported on the ESP8266. The hard part is knowing which questions you need to respond to and what answers you need to give. There’s no source of information that lays all that out clearly, so I had to piece it all together from a number of sources. For some things, I just had to start sending out test questions and then use wireshark to look at the answers being given back.

I work at Apple, so everything in my house is Apple compatible. And Bonjour was invented by Apple, so every Apple product speaks Bonjour. Lots of non-Apple products, like printers and TV sets speak it to. So when I started sending out questions there was no shortage of answers. It took quite a while, but I finally figured out the correct responses and where in the response packets the answers had to go.

Note: Microsoft and Android don’t speak mDNS out of the box, so if you own one of those devices you’ll have to try one of the 3rd party products that can do it for you. 

Bonjour does two things that are interesting to me. The first is called Announcement. This is where a questioner asks “what is the IP address of yardweather.local?”. And you need to respond with the IP address of your device. This is simple and most of the existing implementations get this right. The harder part is Discovery, This is where the questioner asks “what device supports the http service” and any device that does responds with its IP address, the port it’s listening on and other relevant information. This is a function that none of the implementations supported correctly. Most didn’t even try and the ones that did either had malformed packets (as reported by Wireshark) or sent them in the wrong section of the response, so the Bonjour implementation on the Mac didn’t recognize them.

Starting over

I knew I wanted to write my own fresh mDNS implementation. I just needed a starting point. There are many ESP8266 toolkits and many of them have mDNS built in. Even the official SDK from Espressif has one. And there are a few standalone implementations out there. So there was plenty of code to look at. And it was mostly useful. Even though none of the code was fully functional, I saw all the basics of packet encoding and decoding. So I took what I saw and made my own MDNSResponder class. It lets you setup a hostname (so <hostname>.local works). Then it lets you create one or more services. This is where you say you support the http service on port 80, for instance. In my case I wanted to respond that I support a custom protocol for talking to the ESP8266 file system, so I could talk to my IoT devices from a macOS app I am writing.

The code is available on github as part of m8rscript, my custom scripting language made specifically for the ESP8266. The code is still pretty fresh, but already it’s paying dividends. I wrote a m8rsim macOS app which listens for mDNS events. If a ESP9266 with my code is running when m8rsim starts up it will send a PTR response, which macOS recognizes and can talk to the ESP8266 using its Bonjour name. Not only that, but m8rsim is always listening, so if another ESP8266 comes online it will announce itself and will get added the list of known devices. It really makes ESP8266 development much easier than ever before for me.

Stay tuned for move developments with my m8rscript/m8rsim system

m8rScript

The Internet of Things is really taking off. The ESP8266 has made it possible to add a powerful process with wifi to any project for a couple of bucks. Whether you doing a one off art project, or are building a home automation gizmo that you plan on funding with Kickstarter, the ESP8266 can be at the core.

I’ve gotten really excited about this new chip and it’s potential, so naturally I decided to write my own scripting language. When you say that sort of thing in the circles I run around in or when you read it in ESP8266 forums, you generally get a lot of push back. Once the rolling of the eyes and general incredulity settle down, you are likely to get any combination of these 3 responses:

  1. Anyone who writes a new scripting language for any reason on any platform is nuts. There’s nothing you could possibly do that hasn’t already been done.
  2. The ESP8266 already has support for many scripting languages already, based on excellent memory efficient and powerful standards (Javascript, Python, Lua, Basic). Trying to add something new, especially a language not based on an accepted standard, is just nuts.
  3. Scripting languages on a platform like ESP8266 incur useless overhead. Native code FTW. You’re nuts.

TL;DR I’m nuts.

So what motivation could I possibly have for doing this? I’ll push back on each of the points above:

  1. There are plenty of reasons for writing a custom language specific to a particular platform. In the case of the ESP8266, it is both much more powerful that other solutions that have come before it (like the AVR chip in Arduino) and much less powerful than traditional computers for which most of these languages were written. So it makes sense to make a language that enhances its advantages while minimizing its constraints.
  2. All the languages currently on the ESP8266 were originally written for more powerful platforms. Even though the versions being ported are generally “micro” versions, they still push the limits of the platform. For instance, they typically leave less than 25KB of ram storage, It’s hard to do anything in that space. And they generally use garbage collection that adds even more overhead to the management of that tiny memory space.
  3. Native programming is fine as long as you want to upload new code every time you make a change. You might think this problem is solved with OTA. But how do you test your changes without crashing the device, which would require you to bring it back to the shop, crack it open and flash new firmware over the serial port. OTA was designed for mass deployment, not experimentation.

One of the great advantages of m8rscript is that I wrote it to be cross-platform. It runs on both ESP8266 and Mac. That has allowed me to write a macOS app which can simulate m8rscript code before deploying on the device.

To be continued…

Remote Access Howto

Here’s the situation. You installed this cool sprinkler controller. You used a DHCP reservation so it’s always at the same ip address. Now you can always access it – as long as you’re connected to your home network. What if you’re in Hawaii?

IMG_1869

You’re sitting on the beach, sipping a nice drink from a coconut, and suddenly you realize that you just planted a new garden back home that really needs some extra watering. How do you control your sprinklers then?

What you need is the ability to remotely access your home network.

Before we go any further, realize that if you can access your home network from anywhere in the world, so can anyone else. So practice  safe web and always have good passwords on anything accessible from the outside. And when I say that I don’t mean that you should use password as your password (or admin or 12345678). Use a good password, like horse trolley dog boot (although now you shouldn’t use that either!).

To do this you need two things. First, you need a way to get the ip address of your home network router. Most home networks these days use a dynamic ip address. Your ISP changes your address from time to time, similar to the way the DHCP can change the ip of its clients. So you need a dynamic DNS service to associate a domain name with an ip address. That way the address can always be found from the domain name.

There are many services that do this. Some are free (like NoIP), but they usually have pretty severe limitations. So I opted for a paid service. I chose DynDNS (which also has a free service) because they can automatically update the ip address associated with a domain name by communicating with a variety of routers, including an Airport Extreme, which is what I have.

Setup is simple, and well explained on the DynDNS site. You want their Remote Access service. For $25 a year they will give you a domain name, which looks like <your name>.dyndns.org and will keep it updated with whatever your router’s current ip address is. When you sign up they will take you through the steps to setup your router and then you can access your home network with a simple domain name from anywhere in the world. You can even bring your own domain name (which can often be bought for under $10 a year) and use that instead of one of theirs.

Ok, you can get to your router, now what? Well, most routers (hopefully yours) blocks all incoming traffic. That’s to keep the computers inside your network from becoming some hacker’s plaything. But your router can let you open certain ports and redirect that traffic to a certain machine on your network. for instance if you want to login using ssh, you would open port 22 and redirect it to whatever computer you want to login to. As long as that machine has a strong password, you should be pretty safe from attack.

I wanted to access my Open Sprinkler PI server from anywhere in the world, so let’s use that as an example. By default, the OSPI server listens on port 8080, so that’s a reasonable port to open on the router. For an Airport Extreme, you open the Airport Utility app and do the following:

  1. Click on the picture of your router, press Edit and select the Network tab.
  2. Press the + button in the Port Settings: section.
  3. Make sure the Firewall Entry Type: is IPv4 Port Mapping and give your new setting a nice description.
  4. Set the Public and Private TCP Ports to 8080.
  5. Set the private IP Address to your OSPI’s ip address (configured as described here).
  6. Press Save and then Update.

When you router finishes booting up, you should be able to access that service from anywhere. For the above example try typing the URL in a web browser:

http://<your name>.dyndns.org>:8080

and you should see your OSPI control panel.

As a final note. Please be careful opening your network to the outside world. Hackers are relentless and will stop at nothing to compromise your system. So open ports carefully and protect any machines listening on those ports with strong passwords. The ssh port is an especially attractive attack vector. If you want to ssh to your computer, you can get a little extra safety by setting the Public TCP Port to some non-standard number (choose a number between 1024 and 65535) and only set the Private TCP Port to 22. Most ssh clients you use in the world can specify a non-standard port and that will make it harder for hackers to sniff out your ports and commence their attacks.

Have fun with your newfound worldwide power!

DHCP Reservations Howto

I recently installed an Open Sprinkler PI system in my house. This is a great system from Ray’s Hobby which has its own web server. You can access it from a web browser or with an iPhone app which talks to the same server. But you need to access the web server using an ip address, and the easiest way to set the system up is using DHCP, which assigns its own ip addresses. The only way to know what address the OSPI was given is to look at your router’s DHCP table, which is a giant hassle.

That’s what DHCP reservations are for. Every router is different, but most have the ability to assign a consistent ip address in the DHCP range to a given device on the network. Here’s how it’s done using an Airport Extreme and Airport Utility:

  1. Open Airport Utility.
  2. Click on the picture of your router.
  3. Find the wireless client called ospi (you might have to scroll)
  4. Roll over that name and you’ll see a popup.
  5. Write down the hardware address in that popup.
  6. Click on edit and then the Network tab.
  7. Click on + in the DHCP Reservations section.
  8. Give it a nice description and Reserve Address By: Mac Address.
  9. Type in the hardware address you wrote down (including the colons)
  10. There will be an IPv4 address entered, you can keep that or change it. Either way, that will be your reserved ip address.
  11. Hit Save then Update, wait for the router to reboot and your all set!

Now you can enter that ip address in your mobile app and always be able to access your OSPI.

That all works fine as long as you’re connected to your wireless home network. But what if you want to control your sprinklers from anywhere in the world? Well, that’s the topic of another post.

Etherclock

I’ve been planning on building an ethernet connected clock for some time now. It started as a project for my daughter, Mikayla. But my wife went out and bought both her and my other daughter Leah little LED alarm clocks for $10 at Walgreen’s which took away my ability to make her a $50 clock 🙂

Not to worry. We always need more clocks. So I repurposed Etherclock for the family room. We had a Squeezebox serving the purpose of a really accurate clock in that room. But I felt bad to be putting it to such a meager use and my friend Jon wanted a second one, so I sold it to him and started on Etherclock.

The Hardware

First I chose the Tuxgraphics ethernet board. This is a nice, small board that has a mega168 and an ENC28J60 for ethernet. The ethernet chip runs at 25MHz and can output a clock signal which is 1/2 that, which drives the mega168 at 12.5MHz. That’s slower than it’s maximum rate of 16MHz, but it was plenty fast enough for this project. And it was an easy matter to get a precise 1sec time reference using the 16 bit timer/counter of the AVR. Plus it has a nice little prototyping area on one side, and an LED for some early stage debugging.

The Tuxgraphics ethernet board

I wanted the clock to have nice big digits, so I decided on 0.8″ red 7 segment LEDs (LSD8161). I wanted them to be bright enough and have nice even brightness, so I opted for a shift register with constant current outputs. The MAX6969 is a great part for this. It has 16 outputs, so I only needed 2 parts to drive 4 digits, and I didn’t have to worry about multiplexing. I just needed 4 pins: data, clock, enable and latch.

Testing the MAX6969 LED driver

I set the current on the MAX6969 to maximum and used the enable signal to adjust the brightness. I breadboarded one digit to test the serial interface and brightness control. It was convenient to use my STK500 board for the testing phase, using the same mega168 at the same frequency as the final project. I also used this setup to test the IS474 light detector, used to automatic brightness adjustment. The IS474 outputs an analog signal from 0 to 5v, which I brought into the ADC0 pin of the Mega168. The incoming light level determines the percentage of time the LEDs are on, so they’re brighter in a bright room, and get dim when the room light decreases.

The Software

I knew I wanted to try writing code for Etherclock in C++, partly because I have used it for many years at work and partly because I read so many AVR articles claiming it was a bad idea 🙂 So I started writing Marrinator, a C++ library to interface with all the internal and external hardware used in the project. For a while I even toyed with writing my own toolchain, but things have gotten better since then and I abandoned that project. I made up my own Makefile and was running everything from the command line for a while. But I finally settled on using Xcode for development. It is perfectly capable of using an external Makefile and toolchain for building and made it much easier to find and fix compile errors.

Just for fun I played around with the Arduino development system a bit. It’s not hard to get Arduino to work with non-standard boards. But I found that the libraries and development system didn’t really give me much from my roll-your-own approach, so I went back to Xcode.

Getting firmware to the board

When I started the project I was using an STK500 both for hardware prototyping an downloading the firmware to the chip. When it came time to move to the real hardware I had a problem. The STK500 is perfectly capable of programming external hardware, using the standard 6 pin ICSP connector. But the Tuxgraphics board has a 5 pin inline connector instead. So I built what I called The Abomination to interface the two.

The Abomination

Now, on the face of it, it’s not so bad. It’s just a 6 pin IDC to 5 pin inline adaptor, embedded in epoxy. And this worked fine with the STK500. But later on I bought a Dragon, thinking I would be able to use DebugWire for some hardware debugging. That hasn’t happened (yet), but I also started using the Dragon to program the Tuxgraphics board. Of course, that didn’t work. Just look at The Abomination. 6 pin IDC to 5 pin inline. Where’s the missing pin? Turns out it carries VCC, and the Tuxgraphics board has no connection for it. That was fine with the STK500, but the Dragon wants to see VCC to know what voltage it should be programming at. No VCC, no programming.

So I simply added a 6 pin IDC connector to the prototyping area of the Tuxgraphics board, brought in the 5 ICSP pins, added in VCC and was in business.

Tuxgraphics board with the newly added 6 pin ICSP connector. Also notice the connectors for the display board and the button

Note to board designers. If you don’t have a compact board (like the LilyPad or FIO) please add a standard ICSP connector 🙂

Display Board

I put the 2 MAX6969 chips, the IS474 light detector and the 7 segment LEDs on a separate display board, mounted at a right angle to the main board. This made for a nice compact design which didn’t need any additional mounting for the display.

The finished display board. The Magic Mouse is NOT part of the design 🙂

The parts were mounted on a piece of perfboard, which meant soldering. Lots of soldering.

The rat’s nest that is the back of the display board

I used to really like soldering. But display boards like this are an entirely different experience. It consists of around 50 wires, or 100 connections. Shockingly, it worked perfectly the first time I wired it up, which is clearly a tribute to steady hand and high quality tools – or the beer.

Finishing the hardware

The only other parts and connections were a button on top of the case and the power connection at the rear. I didn’t use a polarized connector for power, but I will always try to do that in the future. I haven’t fried anything (yet), but I worry every time I connect power.

The power jack with its scary unpolarized connector
The button mounted in the top. Always use a connector for something like this, so you can disassemble it easily.

The back panel has cutouts for power and the ethernet connector. For ethernet I drilled a round hole and then used the dremel and a file to square it the best I could. The cable fits fine, so I didn’t worry about the look too much.

The case is a Context Engineering split body aluminum box. This is basically 2 pieces of extruded aluminum that fit together, so you can cut them to any length. I cut it to the perfect size with a band saw, which was easy once I had the right blade! But you might notice in the picture above that the cut is a little ragged. Next time I’ll add a fence using a piece of wood and a couple of clamps. I did this for the plexiglass front and it came out perfect.

The perfectly cut plexiglass front

Once the hardware was done, I just had to write all the software, which I’ll discuss in a separate article.

Etherclock in its new home, next to the Apple TV