Putting Data in ESP8266 Flash Memory

The ESP8266 has precious little RAM. You start with a little over 80KB, but the system takes up over 20K of that just for wifi and other required functionality. So best case you get under 60K of RAM for your use. In fact, if you use the Arduino toolchain you’ll be lucky to be left with 30K.

Lots of things you wouldn’t expect use up that RAM. If you’re using C++ and have virtual classes, each one will have a vtable, a little hidden piece of RAM that has pointers to virtual functions. That doesn’t mean you shouldn’t use C++ or even virtual functions. You just need to take care to minimize the number of virtual functions you use.

String constants are typically the biggest offenders. Since they’re constants, you’d think they would go into Flash. But on the Esp Flash memory has access restrictions. You have to access memory on 4 byte boundaries. Attempting to access the bytes of a string stored in Flash would cause a crash. Other data structures like constant arrays used for initialization go into RAM as well. If you’re not careful, you’ll use up all that precious RAM before executing a line of code.

It’s easy to specify that a string or other data structure go into Flash, but I’ve seen a lot of confusion about how to access this memory, so I’ll try to clarify.

Getting your data into Flash memory

The ESP compiler allows you to specify which section a data value should go into. For instance adding this global:

int8_t value __attribute__((section(".irom.text"))) = 'A';

will place that byte in Flash memory. If you then try to access this value you will quickly crash. So what good is it? That’s where accessors come in. Before you continue let me say that you actually need to worry about one more thing. The above attribute will put the value in Flash, but it won’t necessarily align it, so even with an accessor you might get the wrong byte. Another attribute:

__attribute__((aligned(4)))

needs to be added. Now you can access that byte by reading a 32 bit value at its address, which will not crash, and then pick out the first byte and discard the rest. This can get pretty messy pretty quickly. But if you have a good set of accessors and macros, you can save lots of RAM and have good looking code.

The Arduino toolchain makes it pretty easy to put strings in memory. But that’s all derived from its AVR origins, which makes it pretty restrictive. The ESP SDK itself has a very cool feature to help with this. If you turn it on with:

#define USE_OPTIMIZE_PRINTF

then every time you use:

os_printf("This is a nice string\n");

That string will go into Flash. It does this by creating a static variable with the attribute shown above. And os_printf in the SDK will first read the passed string into a temporary RAM buffer using 4 byte aligned reads and output that. It’s nice and simple, but how do you extend that to work elsewhere?

A set of Flash accessors

My m8rscript library has a full set of Flash accessors. I put them directly into my library because those in the SDK are a confusing hodgepodge. There are 2 macros which add the appropriate attributes:

#define RODATA_ATTR  __attribute__((section(".irom.text"))) __attribute__((aligned(4)))
#define ROMSTR_ATTR  __attribute__((section(".irom.text.romstr"))) __attribute__((aligned(4)))

You use the latter for all your strings and the former for everything else. Why two? I found that there are cases where trying to put strings in flash with other data types fails because the compiler can’t figure out how to align them. So I added a new section and added it to my own custom .ld file. This places all the strings together so they can be properly aligned.

Now you can go:

int8_t value RODATA_ATTR = 'A';

And the value will go into Flash and be properly aligned. There’s also an accessor function. You can do this:

int8_t v = readRomByte(&value);

And you’ll get your value without any crashes. You can extend this to byte arrays and have a convenient way to store and access large pieces of data without using up all your RAM.

There are other accessors specifically for strings. You can store a string like this:

const char* myString = ROMSTR("This is a string in Flash");

And then access it like this:

const char* s = new char[ROMstrlen(myString)];
ROMCopyString(s, myString);
<use the string>
delete [ ] s;

The m8rscript library has functions for ROMmemcpy and ROMstrcmp as well. And much of the m8rscript functions allow a const char* strings in Flash. For instance, when you turn a string into an Atom, that string can be in ROM. And the print functions in SystemInterface can take format strings in ROM. That makes it pretty easy to keep strings from eating up RAM.

There is a caveat here. Accessing Flash is much slower that RAM. If you have a function that uses a table and does expensive computations or iterating you need to be careful. Moving the table into Flash could drastically reduce performance. You might want to keep that table in RAM or at least cache parts of it during the expensive parts of your function. The m8rscript library has chosen space over performance and for the most part the perf impact is small.

But what about structures that aren’t bytes? You can access structures in Flash with no problem as long as they are 4 byte aligned. So you can define an array of structures like this:

struct Entry { uint32_t id; uint32_t value; };
Entry RODATA_ATTR myArray [ ] = {
    { 0, 100 },
    { 1, 200 },
};

And now you can access that data without any special accessors. That’s because the elements in the struct are both 4 bytes and the RODATA_ATTR aligns the whole array on 4 byte boundaries.

With aggressive use of this technique, m8rscript uses just a bit over 30K of RAM, leaving over 50K for your scripts. So use these techniques and go save some RAM!

 

Multicast DNS Tribulations

For my m8rScript project I really want to use Bonjour. It would give me a way of finding many ESP8266 devices on the network without having to nail down IP addresses. The concept is simple: you give your device a name, like yardweather. Then you type yardweather.local into your web browser and up pops the web page for your device. Additionally, you can broadcast a message on your network asking if any device supports a given service, and any device that does sends back information telling you about itself. So you can know how you are allowed to talk to it. Simple, right?

NO

I tried lots of options for the ESP8266. BUT NONE OF THEM WORKED! To make matters worse The documentation is terrible. In fact, maybe the reason none of the implementations work is because of the bad documentation.

Ok, I’m being a little unfair. All of the implementations work somewhat, but none have all the features to let you achieve the Nirvana of being able to both find out how to contact the device and what the device is capable of. So I started on a trek, first to understand what mDNS is all about and second to write an implementation for the ESP8266.

What is mDNS?

It goes by many names: Multicast DNS is it’s official IETF name, as documented here. Apple invented it and they call it Bonjour. They originally wanted to call it Rendezvous, but that name was taken, so they switched to the equally French Bonjour. And all this falls under the umbrella of Zero Configuration Networking (zeroconf).

When I started looking around I would search for “mdns packet format” and things like that and it would keep coming back with DNS hits. It didn’t occur to me at the time that mDNS IS actually a particular flavor of the Domain Name System (DNS), which is pretty much the backbone of the web. With that new understanding, a lot of the terminology and descriptions made a lot more sense. You ask a question (“what is the IP address for this domain name“) and get back an answer (“1.2.3.4”). Then you can contact that IP address and start a conversation. The two interesting bits about mDNS are a) the types of questions you ask and the responses you get back, and b) the fact that it uses multicast. Normally you ask DNS questions of a server with a particular IP address, which is added to your computer when it is first set up. Multicast just sends its questions out into the universe and then waits a while for answers to come back, often from many different sources.

For the ESP8266 the multicast part is easy. It uses a networking technology called UDP, which is well supported on the ESP8266. The hard part is knowing which questions you need to respond to and what answers you need to give. There’s no source of information that lays all that out clearly, so I had to piece it all together from a number of sources. For some things, I just had to start sending out test questions and then use wireshark to look at the answers being given back.

I work at Apple, so everything in my house is Apple compatible. And Bonjour was invented by Apple, so every Apple product speaks Bonjour. Lots of non-Apple products, like printers and TV sets speak it to. So when I started sending out questions there was no shortage of answers. It took quite a while, but I finally figured out the correct responses and where in the response packets the answers had to go.

Note: Microsoft and Android don’t speak mDNS out of the box, so if you own one of those devices you’ll have to try one of the 3rd party products that can do it for you. 

Bonjour does two things that are interesting to me. The first is called Announcement. This is where a questioner asks “what is the IP address of yardweather.local?”. And you need to respond with the IP address of your device. This is simple and most of the existing implementations get this right. The harder part is Discovery, This is where the questioner asks “what device supports the http service” and any device that does responds with its IP address, the port it’s listening on and other relevant information. This is a function that none of the implementations supported correctly. Most didn’t even try and the ones that did either had malformed packets (as reported by Wireshark) or sent them in the wrong section of the response, so the Bonjour implementation on the Mac didn’t recognize them.

Starting over

I knew I wanted to write my own fresh mDNS implementation. I just needed a starting point. There are many ESP8266 toolkits and many of them have mDNS built in. Even the official SDK from Espressif has one. And there are a few standalone implementations out there. So there was plenty of code to look at. And it was mostly useful. Even though none of the code was fully functional, I saw all the basics of packet encoding and decoding. So I took what I saw and made my own MDNSResponder class. It lets you setup a hostname (so <hostname>.local works). Then it lets you create one or more services. This is where you say you support the http service on port 80, for instance. In my case I wanted to respond that I support a custom protocol for talking to the ESP8266 file system, so I could talk to my IoT devices from a macOS app I am writing.

The code is available on github as part of m8rscript, my custom scripting language made specifically for the ESP8266. The code is still pretty fresh, but already it’s paying dividends. I wrote a m8rsim macOS app which listens for mDNS events. If a ESP9266 with my code is running when m8rsim starts up it will send a PTR response, which macOS recognizes and can talk to the ESP8266 using its Bonjour name. Not only that, but m8rsim is always listening, so if another ESP8266 comes online it will announce itself and will get added the list of known devices. It really makes ESP8266 development much easier than ever before for me.

Stay tuned for move developments with my m8rscript/m8rsim system

The Internet of Things

If you have even one nerd bone in your body, you’ve heard about the Internet of Things or IoT. It’s all the rage. There used to be a joke about connecting your toaster to the internet. It was a metaphor for just how crazy things might get someday if left unchecked. Well, that day is here

Posted in IoT

m8rScript

The Internet of Things is really taking off. The ESP8266 has made it possible to add a powerful process with wifi to any project for a couple of bucks. Whether you doing a one off art project, or are building a home automation gizmo that you plan on funding with Kickstarter, the ESP8266 can be at the core.

I’ve gotten really excited about this new chip and it’s potential, so naturally I decided to write my own scripting language. When you say that sort of thing in the circles I run around in or when you read it in ESP8266 forums, you generally get a lot of push back. Once the rolling of the eyes and general incredulity settle down, you are likely to get any combination of these 3 responses:

  1. Anyone who writes a new scripting language for any reason on any platform is nuts. There’s nothing you could possibly do that hasn’t already been done.
  2. The ESP8266 already has support for many scripting languages already, based on excellent memory efficient and powerful standards (Javascript, Python, Lua, Basic). Trying to add something new, especially a language not based on an accepted standard, is just nuts.
  3. Scripting languages on a platform like ESP8266 incur useless overhead. Native code FTW. You’re nuts.

TL;DR I’m nuts.

So what motivation could I possibly have for doing this? I’ll push back on each of the points above:

  1. There are plenty of reasons for writing a custom language specific to a particular platform. In the case of the ESP8266, it is both much more powerful that other solutions that have come before it (like the AVR chip in Arduino) and much less powerful than traditional computers for which most of these languages were written. So it makes sense to make a language that enhances its advantages while minimizing its constraints.
  2. All the languages currently on the ESP8266 were originally written for more powerful platforms. Even though the versions being ported are generally “micro” versions, they still push the limits of the platform. For instance, they typically leave less than 25KB of ram storage, It’s hard to do anything in that space. And they generally use garbage collection that adds even more overhead to the management of that tiny memory space.
  3. Native programming is fine as long as you want to upload new code every time you make a change. You might think this problem is solved with OTA. But how do you test your changes without crashing the device, which would require you to bring it back to the shop, crack it open and flash new firmware over the serial port. OTA was designed for mass deployment, not experimentation.

One of the great advantages of m8rscript is that I wrote it to be cross-platform. It runs on both ESP8266 and Mac. That has allowed me to write a macOS app which can simulate m8rscript code before deploying on the device.

To be continued…