Putting Data in ESP8266 Flash Memory

The ESP8266 has precious little RAM. You start with a little over 80KB, but the system takes up over 20K of that just for wifi and other required functionality. So best case you get under 60K of RAM for your use. In fact, if you use the Arduino toolchain you’ll be lucky to be left with 30K.

Lots of things you wouldn’t expect use up that RAM. If you’re using C++ and have virtual classes, each one will have a vtable, a little hidden piece of RAM that has pointers to virtual functions. That doesn’t mean you shouldn’t use C++ or even virtual functions. You just need to take care to minimize the number of virtual functions you use.

String constants are typically the biggest offenders. Since they’re constants, you’d think they would go into Flash. But on the Esp Flash memory has access restrictions. You have to access memory on 4 byte boundaries. Attempting to access the bytes of a string stored in Flash would cause a crash. Other data structures like constant arrays used for initialization go into RAM as well. If you’re not careful, you’ll use up all that precious RAM before executing a line of code.

It’s easy to specify that a string or other data structure go into Flash, but I’ve seen a lot of confusion about how to access this memory, so I’ll try to clarify.

Getting your data into Flash memory

The ESP compiler allows you to specify which section a data value should go into. For instance adding this global:

int8_t value __attribute__((section(".irom.text"))) = 'A';

will place that byte in Flash memory. If you then try to access this value you will quickly crash. So what good is it? That’s where accessors come in. Before you continue let me say that you actually need to worry about one more thing. The above attribute will put the value in Flash, but it won’t necessarily align it, so even with an accessor you might get the wrong byte. Another attribute:


needs to be added. Now you can access that byte by reading a 32 bit value at its address, which will not crash, and then pick out the first byte and discard the rest. This can get pretty messy pretty quickly. But if you have a good set of accessors and macros, you can save lots of RAM and have good looking code.

The Arduino toolchain makes it pretty easy to put strings in memory. But that’s all derived from its AVR origins, which makes it pretty restrictive. The ESP SDK itself has a very cool feature to help with this. If you turn it on with:


then every time you use:

os_printf("This is a nice string\n");

That string will go into Flash. It does this by creating a static variable with the attribute shown above. And os_printf in the SDK will first read the passed string into a temporary RAM buffer using 4 byte aligned reads and output that. It’s nice and simple, but how do you extend that to work elsewhere?

A set of Flash accessors

My m8rscript library has a full set of Flash accessors. I put them directly into my library because those in the SDK are a confusing hodgepodge. There are 2 macros which add the appropriate attributes:

#define RODATA_ATTR  __attribute__((section(".irom.text"))) __attribute__((aligned(4)))
#define ROMSTR_ATTR  __attribute__((section(".irom.text.romstr"))) __attribute__((aligned(4)))

You use the latter for all your strings and the former for everything else. Why two? I found that there are cases where trying to put strings in flash with other data types fails because the compiler can’t figure out how to align them. So I added a new section and added it to my own custom .ld file. This places all the strings together so they can be properly aligned.

Now you can go:

int8_t value RODATA_ATTR = 'A';

And the value will go into Flash and be properly aligned. There’s also an accessor function. You can do this:

int8_t v = readRomByte(&value);

And you’ll get your value without any crashes. You can extend this to byte arrays and have a convenient way to store and access large pieces of data without using up all your RAM.

There are other accessors specifically for strings. You can store a string like this:

const char* myString = ROMSTR("This is a string in Flash");

And then access it like this:

const char* s = new char[ROMstrlen(myString)];
ROMCopyString(s, myString);
<use the string>
delete [ ] s;

The m8rscript library has functions for ROMmemcpy and ROMstrcmp as well. And much of the m8rscript functions allow a const char* strings in Flash. For instance, when you turn a string into an Atom, that string can be in ROM. And the print functions in SystemInterface can take format strings in ROM. That makes it pretty easy to keep strings from eating up RAM.

There is a caveat here. Accessing Flash is much slower that RAM. If you have a function that uses a table and does expensive computations or iterating you need to be careful. Moving the table into Flash could drastically reduce performance. You might want to keep that table in RAM or at least cache parts of it during the expensive parts of your function. The m8rscript library has chosen space over performance and for the most part the perf impact is small.

But what about structures that aren’t bytes? You can access structures in Flash with no problem as long as they are 4 byte aligned. So you can define an array of structures like this:

struct Entry { uint32_t id; uint32_t value; };
Entry RODATA_ATTR myArray [ ] = {
    { 0, 100 },
    { 1, 200 },

And now you can access that data without any special accessors. That’s because the elements in the struct are both 4 bytes and the RODATA_ATTR aligns the whole array on 4 byte boundaries.

With aggressive use of this technique, m8rscript uses just a bit over 30K of RAM, leaving over 50K for your scripts. So use these techniques and go save some RAM!


Multicast DNS Tribulations

For my m8rScript project I really want to use Bonjour. It would give me a way of finding many ESP8266 devices on the network without having to nail down IP addresses. The concept is simple: you give your device a name, like yardweather. Then you type yardweather.local into your web browser and up pops the web page for your device. Additionally, you can broadcast a message on your network asking if any device supports a given service, and any device that does sends back information telling you about itself. So you can know how you are allowed to talk to it. Simple, right?


I tried lots of options for the ESP8266. BUT NONE OF THEM WORKED! To make matters worse The documentation is terrible. In fact, maybe the reason none of the implementations work is because of the bad documentation.

Ok, I’m being a little unfair. All of the implementations work somewhat, but none have all the features to let you achieve the Nirvana of being able to both find out how to contact the device and what the device is capable of. So I started on a trek, first to understand what mDNS is all about and second to write an implementation for the ESP8266.

What is mDNS?

It goes by many names: Multicast DNS is it’s official IETF name, as documented here. Apple invented it and they call it Bonjour. They originally wanted to call it Rendezvous, but that name was taken, so they switched to the equally French Bonjour. And all this falls under the umbrella of Zero Configuration Networking (zeroconf).

When I started looking around I would search for “mdns packet format” and things like that and it would keep coming back with DNS hits. It didn’t occur to me at the time that mDNS IS actually a particular flavor of the Domain Name System (DNS), which is pretty much the backbone of the web. With that new understanding, a lot of the terminology and descriptions made a lot more sense. You ask a question (“what is the IP address for this domain name“) and get back an answer (“”). Then you can contact that IP address and start a conversation. The two interesting bits about mDNS are a) the types of questions you ask and the responses you get back, and b) the fact that it uses multicast. Normally you ask DNS questions of a server with a particular IP address, which is added to your computer when it is first set up. Multicast just sends its questions out into the universe and then waits a while for answers to come back, often from many different sources.

For the ESP8266 the multicast part is easy. It uses a networking technology called UDP, which is well supported on the ESP8266. The hard part is knowing which questions you need to respond to and what answers you need to give. There’s no source of information that lays all that out clearly, so I had to piece it all together from a number of sources. For some things, I just had to start sending out test questions and then use wireshark to look at the answers being given back.

I work at Apple, so everything in my house is Apple compatible. And Bonjour was invented by Apple, so every Apple product speaks Bonjour. Lots of non-Apple products, like printers and TV sets speak it to. So when I started sending out questions there was no shortage of answers. It took quite a while, but I finally figured out the correct responses and where in the response packets the answers had to go.

Note: Microsoft and Android don’t speak mDNS out of the box, so if you own one of those devices you’ll have to try one of the 3rd party products that can do it for you. 

Bonjour does two things that are interesting to me. The first is called Announcement. This is where a questioner asks “what is the IP address of yardweather.local?”. And you need to respond with the IP address of your device. This is simple and most of the existing implementations get this right. The harder part is Discovery, This is where the questioner asks “what device supports the http service” and any device that does responds with its IP address, the port it’s listening on and other relevant information. This is a function that none of the implementations supported correctly. Most didn’t even try and the ones that did either had malformed packets (as reported by Wireshark) or sent them in the wrong section of the response, so the Bonjour implementation on the Mac didn’t recognize them.

Starting over

I knew I wanted to write my own fresh mDNS implementation. I just needed a starting point. There are many ESP8266 toolkits and many of them have mDNS built in. Even the official SDK from Espressif has one. And there are a few standalone implementations out there. So there was plenty of code to look at. And it was mostly useful. Even though none of the code was fully functional, I saw all the basics of packet encoding and decoding. So I took what I saw and made my own MDNSResponder class. It lets you setup a hostname (so <hostname>.local works). Then it lets you create one or more services. This is where you say you support the http service on port 80, for instance. In my case I wanted to respond that I support a custom protocol for talking to the ESP8266 file system, so I could talk to my IoT devices from a macOS app I am writing.

The code is available on github as part of m8rscript, my custom scripting language made specifically for the ESP8266. The code is still pretty fresh, but already it’s paying dividends. I wrote a m8rsim macOS app which listens for mDNS events. If a ESP9266 with my code is running when m8rsim starts up it will send a PTR response, which macOS recognizes and can talk to the ESP8266 using its Bonjour name. Not only that, but m8rsim is always listening, so if another ESP8266 comes online it will announce itself and will get added the list of known devices. It really makes ESP8266 development much easier than ever before for me.

Stay tuned for move developments with my m8rscript/m8rsim system

The Internet of Things

If you have even one nerd bone in your body, you’ve heard about the Internet of Things or IoT. It’s all the rage. There used to be a joke about connecting your toaster to the internet. It was a metaphor for just how crazy things might get someday if left unchecked. Well, that day is here

Posted in IoT


The Internet of Things is really taking off. The ESP8266 has made it possible to add a powerful process with wifi to any project for a couple of bucks. Whether you doing a one off art project, or are building a home automation gizmo that you plan on funding with Kickstarter, the ESP8266 can be at the core.

I’ve gotten really excited about this new chip and it’s potential, so naturally I decided to write my own scripting language. When you say that sort of thing in the circles I run around in or when you read it in ESP8266 forums, you generally get a lot of push back. Once the rolling of the eyes and general incredulity settle down, you are likely to get any combination of these 3 responses:

  1. Anyone who writes a new scripting language for any reason on any platform is nuts. There’s nothing you could possibly do that hasn’t already been done.
  2. The ESP8266 already has support for many scripting languages already, based on excellent memory efficient and powerful standards (Javascript, Python, Lua, Basic). Trying to add something new, especially a language not based on an accepted standard, is just nuts.
  3. Scripting languages on a platform like ESP8266 incur useless overhead. Native code FTW. You’re nuts.

TL;DR I’m nuts.

So what motivation could I possibly have for doing this? I’ll push back on each of the points above:

  1. There are plenty of reasons for writing a custom language specific to a particular platform. In the case of the ESP8266, it is both much more powerful that other solutions that have come before it (like the AVR chip in Arduino) and much less powerful than traditional computers for which most of these languages were written. So it makes sense to make a language that enhances its advantages while minimizing its constraints.
  2. All the languages currently on the ESP8266 were originally written for more powerful platforms. Even though the versions being ported are generally “micro” versions, they still push the limits of the platform. For instance, they typically leave less than 25KB of ram storage, It’s hard to do anything in that space. And they generally use garbage collection that adds even more overhead to the management of that tiny memory space.
  3. Native programming is fine as long as you want to upload new code every time you make a change. You might think this problem is solved with OTA. But how do you test your changes without crashing the device, which would require you to bring it back to the shop, crack it open and flash new firmware over the serial port. OTA was designed for mass deployment, not experimentation.

One of the great advantages of m8rscript is that I wrote it to be cross-platform. It runs on both ESP8266 and Mac. That has allowed me to write a macOS app which can simulate m8rscript code before deploying on the device.

To be continued…

Easy Library Table

LibraryTableCompMy kids are in high school now so I thought it was time they had nicer desks than the ones we got them at Ikea. I wanted to make something that would last, that they could take with them to college and beyond. I also decided I didn’t want a desk with drawers on the side, just one small drawer at the top for pencils and papers. This was both because they already have roll around cabinets for bigger things and because I didn’t want to complicate this project with a lot of drawers, which are harder to build.

library-cWhat I really wanted was not a desk at all, but a Library Table. I’ve always like Craftsman style furniture, so I went looking for plans. The internet is a great place for finding just about anything and I quickly came upon two sets of free plans as a starting point:

Screen Shot 2014-07-10 at 7.31.23 AMScreen Shot 2014-07-10 at 8.04.02 AM






I liked features of both of these desks. But both were pretty complicated, using mortise and tenon joints and wooden drawer slides. I can use a table saw and other power tools, but I’m no master craftsman, so I set out to simplify the plans and come up with something I could do a good job building.


The first thing I needed was a joinery system. I was really worried about this because all the joinery I saw being used in the plans I looked at seemed beyond my abilities. So I did some research and decided on using dowels for the main leg supports, rabbet joints for the drawers and biscuits for everything else.


When it comes to wood there are lots of choices. You want to select an attractive hardwood for durability. But that doesn’t narrow it down much. Here in the San Francisco Bay Area there are many different woods available. If you go down to Southern Lumber you’ll see an incredible selection. But you’ll also see eye popping prices. I did a sample pricing using cherry wood. It would have been a beautiful desk for around $1500 or so. Not only is it expensive, but at that price I would have been terrified with every cut.

redoakSo I went down to Home Depot and found that they had a good supply of red oak. Oak is great wood, if not very exciting. It’s hard, but not very difficult to work with. By contrast, I have some Ipé from a deck I had installed. It is a hardwood too. But it’s super difficult to cut and produces sawdust that’s a real irritant. Your eyes water, your skin itches and you just can’t wait for the whole ordeal to be over.

The oak at Home Depot was reasonably priced. I was able to get everything I needed for around $300. And it had nice straight boards with good grain at a variety of lengths and widths, all 3/4″ thick. They also have maple and cherry (and a little walnut). But they’re more expensive with much less variety and quality of boards. So red oak it was.

My one complaint about the Home Depot boards is that they put one or two stickers on each one. They are paper not plastic, so they rip easily and you really have to work at it to get them off. At first I thought I could just sand them off. But that’s much harder than it seems. So I recommend you get into a Zen place and peel off the stickers when you first buy the boards. The glue left behind is easy to sand off and you’ll be able to pick either side of all boards.

In the plans below I show all the wood and sizes needed. You’ll have to cut these out of various lengths of boards to get what you need. I’ll leave it up to you how you do that. Home Depot has lots of 7′ boards in 3 1/2″ and 5 1/2″ widths. But they have lots of other lengths and widths. If you have a way to transport it, you might be more efficient with a 10 or 12 foot piece of lumber. Just think about what combination of pieces you can get out of each board to minimize waste (both of wood and money).


In the process of doing a few woodworking projects I’ve found that you can never have enough clamps. Seems like every time I start a new project I buy 2 or 3 new clamps. My favorites for long edge clamping are pipe clamps. pipe clampThey are pretty cheap and you buy a piece of metal pipe to make them any length you want. I have 4: 2 using half inch pipe and 2 with 3/4 inch, all 48″ long. They can really apply a lot of pressure and are long enough to clamp the legs to the cross members of the desk.

I also have several C clamps, some bar clamps in 1, 2 and 3 foot lengths, a some smaller (and less useful) spring clamps. Make sure you have enough of each.

Building the Desk

The Top

To make the top, you need to glue together enough boards to get the 24″ width needed, using biscuits for strength. I used all 5 1/2″ stock, 5 board wide, and cut off the extra 3 1/2″. I recommend starting with the top. You want to select the nicest boards, laying them side by side to see which have the most consistent color and grain and flipping them to see which side is nicer. I found that if you do this and use a light colored varnish you can’t really see any seams at all.

The Legs

The legs are 2″ x 2″ and 29″ long. Since all the wood from Home Depot is 3/4″ thick, I glued 3 boards together to get that thickness. Lots of glueing surface, so no need for any joinery Here. I actually glued together 3 – 5 1/2″ x 30″ boards and then cut them lengthwise into 2 – 2″ x 2 1/4″ legs. My plan was to put a veneer on the face of the legs with with the seams. You make the veneer by cutting a 1/8″ wide strip from a 3 1/2″ wide board and glueing a piece of that to each side. But I found I could put the veneer on one of the faces since the other would be facing to the back. That made it easier to cut the legs to exactly 2″ x 2″, without having to worry about the exact thickness of the veneer.

End Pieces

Once the legs are completed it’s time to attach them to the end pieces and the slats. I recommend you sand everything before starting assembly just to make it easier to get into all the corners. The easiest way to proceed is to drill the dowel holes in the top end piece and legs. In fact you’ll want to drill the holes for the cross pieces. That will be 4 holes for the back piece on both sides and two holes for each of the two front pieces on both sides. This will make it a lot easier to do the next step. There’s one last bit of work that needs to be done before assembling the front, so you might as well do it now. Cut the vertical biscuit hole for the right and left front pieces in the front legs. This will be between the two front cross pieces.

Now mark the centers of the attachment points for the slats and cut the biscuit holes for them, as well as for the bottom end piece. Now glue and clamp the slats to the top and bottom pieces and before they are dry, glue and clamp the end pieces to the legs. Assuming all the cuts are straight, everything should be perfectly square. But biscuits have a bit of slop, so make sure the slats like up with the center line marks you made. Set these aside and let them dry, then you’re ready for the next step.

Cross Pieces

Now it’s time to connect the cross pieces, making a structure that will stand on its own. Since you’ve already made the dowel holes, the only cutting here is the biscuit holes for the bottom cross brace in back. Once that’s done you can glue and clamp the two front top cross pieces, the back top cross piece and the back bottom cross piece. Let everything dry and you’re ready for the next step.

Front and Drawer Runners

These pieces fit together and attach to the desk using biscuits. After cutting the biscuit holes, glue the runners to the back then glue in the front right and left pieces. You might want to put a bit of glue on the top and bottom cross pieces, too. But everything should stay pretty solid from here.

The Drawer

The drawer uses locking rabbet joints in the front so it withstands repeated opening, and a standard rabbet in the back. Follow the directions in the link above to cut the slot in the front piece and the locking slots in the front of the side pieces. Then cut a standard 1/2″ x 1/4″ rabbet in the back of the side pieces. Once that’s done, setup the table saw to cut a slot lengthwise 1/4″ from the bottom of all sides and 1/4″ deep. This is for the bottom to slide into. Next glue and nail the sides to the front, then slide in the bottom plywood, then glue and nail the back to the sides. You just made a drawer!

Now is the tricky part. Assuming you got the measurements right, you should have just enough space for the drawer sliders to fit between the drawer sides and the slide supports in the desk. I had to shim a bit, using a couple of washers, to get things to fit just right. Also be careful to make the amount of space above and below the drawer the same. Take your time you should be able to get a nice smooth drawer operation with no rubbing of wood on wood.

Drawer Facade

There is a facade on the front of the drawer to hide the slides. I made a front as a single 3 1/2″ x 36″ piece and then cut it in 3 pieces to get the two side pieces and the facade. That way, the grain matches nicely. But you’ll have to trim the facade so it fits with some clearance. I started by taking 1/16″ off of all 4 sides and continued trimming tiny bits until there was good clearance, but not a big gap.

Next I drilled 2 holes in the front drawer piece (the piece behind the facade). Now I held the facade against the drawer front until there was the same clearance on all 4 sides, marked the positions of the 2 screws, and drilled pilot holes. Don’t drill all the way through the facade. I use blue masking tape on the drill bit 1/2″ down just to make sure. Then screw the facade on with 1″ wood screws and you’re set. Test the fit and do any final adjustments to the slides.

As a last step, drill and install your drawer pull. I used a simple burnished metal knob because it fit the look and only needed one hold to install. But the choice is yours.

Attach the Top

188101You attach the top using screw, not glue, to allow for expansion. I used figure 8 fasteners. If you can find them use the ones specifically designer for desks. They fit wood screws nicely without the need for additional washers. To install them select your locations on the desk frame (I chose 6 locations: 2 each on the front and back and 1 on each end). Then use a Forstner bit of the appropriate size and drill deep enough so the fastener sits flush with the top of the frame. Now go around, drill pilot holes and screw the fasteners into the frame.

UnknownForstner bits are not cheap, but they are really useful. They let you drill holes with flat bottoms and are perfect for this sort of counter sinking work. They’re not cheap, but you can buy them individually. So just get one of the right size for this job and build up your collection later. I bought a couple and then asked for more sizes for Christmas 🙂

Next place the top on the frame and adjust it so it’s even on all 4 sides. Then mark the location of the holes in the fasteners, drill pilot holes and screw the top to the frame. You desk is now built!

Sand and Varnish

Hopefully you’ve been doing some preliminary sanding before putting everything together. If so, you just need to do a light final sanding and then wipe everything down with a damp cloth. Wait a few minutes for it to dry and then put on a coat of polyurethane varnish. A quart is plenty. Let that coat dry, lightly sand the entire desk, and give a pretty good sanding to the top. Then wipe off and apply a second coat. Two coats is good enough, but to get a really smooth top, you can sand and coat only the top once more.

I didn’t actually buy varnish from Amazon. I went down to the hardware store because they have Minwax stain samples. Red oak is a little light and doesn’t have that classic golden oak tone. So I got 4 samples and stained a scrap piece to see which one looked best. I ended up going with Golden Oak (bit surprise!).

Cut List

Library Table Exploded

  • Legs – 2″ x 2″ x 29″ (3 – 2″ x 3/4″ x 29″, cut to side with 1/8″ veneer on one side)
  • Top side – 3/4″ x 5 1/2″ x 17 1/2″
  • Bottom side – 3/4″ x 3 1/2″ x 17 1/2″
  • Side slats – 1/2″ x 3″ x 16″
  • Top – 24″ x 46″ x 3/4″ (3 – 5 1/2″ wide, 2 – 3 3/4″ wide)
  • Top back cross piece – 5 1/2″ x 36″
  • Bottom back cross piece – 3 1/2″ x 36″
  • Front cross pieces – 2 – 2″ x 36″
  • Front facing piece – 3 1/2″ x 36″, cut into 2 – 8″ side pieces and 1 – 20″ facade
  • Drawer slide supports – 2 – 3″ x 19 3/8″ x 3/4″
  • Drawer front – 2 3/4″ x 17″ x 1/2″
  • Drawer back – 2 3/4″ x 17″ x 1/2″
  • Drawer sides – 2 3/4″ x 19″ x 1/2″
  • Drawer bottom – 17″ x  18″ x 1/2″ birch (or any hardwood) plywood, one side appearance grade
  • Drawer slides
  • Drawer knob – to suit your taste

Enjoy Your New Desk!