HN Front Page - April 30
Text Size:   Decrease text size   Increase text size    

How I got an FBI record at age 11 from dabbling in cryptography (2015)

How I got an FBI record at age 11 from dabbling in cryptography then got into more trouble


Les Earnest <les at>


Growing up in San Diego, my first encounter with advanced technology was the gift of a one-speed fat tired bicycle in 1937. The second one, acquired a short time later, was my own radio with mysteriously glowing vacuum tubes, which enabled me to listen to a series of 15 minute kids radio programs every afternoon, such as Magic Island and Jack Armstrong, the All American Boy. 


At some point the Jack Armstrong program invited listeners to mail in a Wheaties box top to get a decoder ring that could be used to decipher secret messages that would be given near the end of certain broadcasts.  I sent for it as did Bobby Bond, my best friend through most of grammar school.  Bobby was particularly intrigued with cryptography and in 1942 he bought a new book called Secret and Urgent. Note that this was early in World War II.  We both read it and learned how to use letter frequencies to break ciphers, then went on to more advance topics. 

Fletcher Pratt, Secret and Urgent, Blue Ribbon Books; Garden City, NY; 1942


Bobby and I decided that we needed to have a secure way to communicate with each other, so we put together a rather elaborate jargon code based on the principles described in the book. I don't remember exactly why we thought we needed it. We spent nearly every afternoon together so there was ample time to talk privately.  Still, you never could tell when you might need to send a secret message!


We typed up the code key on single sheet of paper with a carbon copy and each carried one at all times.  I had recently been given eyeglasses but didn't like to wear them, so I kept them in a hard case in the pocket of my trousers.  I figured that this was a good place to keep my copy of the code key, so I carefully folded it to one-eighth of its original size and stuck it at the bottom of the case, under my glasses.


Every chance I got, I went body surfing at Old Mission Beach and usually went by streetcar. On my way home from the beach one time the case carrying my glasses somehow slipped out of my pocket unnoticed. When I reported the loss to my mother that night, she called the streetcar company to see if it had been turned in. Unfortunately it hadn't.


After a few weeks of waiting in vain for the glasses to turn up, we began to lose hope.  My mother didn't rush to replace them in view of the fact that I hadn't worn them much and they cost about $8, a large sum at that time. (To me, $8 represented 40 round trips to the beach by streetcar, or 80 admission fees to the movies.)


Unknown to us, the case had been found by a patriotic citizen who opened it, discovered the code key, recognized that it must belong to a Japanese spy and turned it over to the FBI.  This was just after local citizens of Japanese descent had been rounded up and taken away to concentration camps, though I was not aware of that at the time.  I remember hearing that a local grocer was secretly a Colonel in the Japanese Army and had hidden his uniform in the back of his store. A lot of people actually believed such things.


About ten weeks later, my mother told me that she received a mysterious telephone call at workat that time she was Vice Principal at Roosevelt Junior High.  The caller, who identified himself as an FBI agent, said, I want an appointment with you at once.  She said, Come right over to my office.  No, we must see you in your home, was the reply. She went home and waited for a substantial period.  I happened to be off on another escapade, so I wasn't aware of all this.


Eventually a black limousine pulled up in front of the house. Two men sat in it reading notes then eventually came up the steps. As my mother let them in the living room, each rolled back his coat lapel to flash identification and said something like, I'm Joe Blow of the FBI.  One of them then threw my glasses on the coffee table and said, Have you seen these before?  My mother replied, They look like my son's glasses, which he lost awhile ago.  They are your son's alright, said one of them.


They wanted to know why there was a code sheet in the case with the glasses.  My mother said we had been studying cryptography and that this was no doubt something that we had put together for fun. At first they refused to believe her, arguing that the code sheet could not have been compiled by kids, but after awhile, one of the two began to be a bit friendlier.


My mother told the investigators how glad she was to get the glasses back, considering that they cost $8. The sourpuss did a slow burn, then said Lady, this case has cost the government thousands of dollars. It has been the top priority in our office for the last eight weeks. We traced the glasses to your son from the prescription by examining the files of all optometrists in the San Diego area.  He went on to say that they had been interviewing our friends and neighbors for several weeks.


The friendlier one eventually described how much it had cost to investigate another recent case where a person was reported to have pulled down an American flag and stepped on it.  Only after the investigation was well under way did they learn that the perpetrator of this nefarious act was only four years old.


The tough agent apparently remained convinced to the end that I really was a Japanese spy.  He insisted on keeping code sheet for our records.  He apparently wanted to be in a position to decode any of our secret communications if they should find any.


Since our coding scheme had been compromised, Bobby and I devised a new key.  I started carrying it in my wallet, which I felt was more secure.  I don't remember ever exchanging any coded messages but I was always ready.  I didnt expect anything more to come of our misadventure but we managed to get into more trouble.


Postscript: Bobby and I got into more trouble

Bobbys dad was a medical doctor and at some point Bobby discovered a stack of preprinted forms for recording information about pregnant women. As an expression of wishful thinking we filled one out in the name of a cute 12-year-old girl who lived across the street from me and who everyone called Alabama because that was where she was from and because she spoke with a charming southern drawl.


After we left the form by her front door her parents somehow figured out who had done that and, when Bobbys and my parents learned of this stunt they decreed that we would no longer play together. We followed that guidance for over 40 years.


A few years later, when I was in college, I ran into Alabama on a warm summer day at Old Mission Beach where I was body surfing. Not surprisingly she was sexier than ever, wearing a bikini bottom but Instead of a bra she had a scarf tied around her top. When I learned that she had come there by public transit I offered her a ride home even though she no longer lived in my neighborhood, which she accepted. We had a friendly chat at the beach and on the way to her home but I didnt follow up by seeking a date, figuring that her parents likely still had a negative view of me.


A slap on the wrist

Around the same time, I got a summer job at the Naval Electronics Lab in San Diego working on the assessment of sonar and underwater passive listening systems. This involved going to sea on a round-bottomed boat that rocked a lot and, after I had chili for lunch the first day, I experienced substantial lossage. I also got to go out on a submarine, which was much cooler, though there was some consternation when I popped up on the conning tower without doing the protocol of asking permission. I noticed that our passive listening system, using microphones on tripods resting on the sea bottom, managed to detect the submarine just once, when it ran into the microphone and made a large thunk.


At some point they decided that I should have a security clearance and one of the questions on the single page application form was Have you ever been investigated by the FBI? Naturally, I checked Yes. The next question was, If so, describe the circumstances. There was very little space on the form, so I answered simply and honestly, I was suspected of being a Japanese spy. When I handed the form in to the security officer, he scanned it quickly, looked me over slowly, then said, Explain this--pointing at the FBI question. I described what had happened.


He then got very agitated, picked up my form, tore it in pieces, and threw it in the waste basket, then handed me a blank form, saying Here, fill it out again and don't mention that. If you do, I'll make sure that you never get a security clearance.


I did as he directed, thus lying, and was shortly granted the clearance. I never again disclosed that incident on security clearance forms. However about twelve years later I learned by chance that putting slightly provocative information on a security clearance form can greatly speed up the clearance process. See my journal article, Can computers cope with human races?


Epilogue: Bobby and I start a new conspiracy

In the early 1970s I was drawn into bicycle racing by my two sons and, after racing a bit, took up officiating, then became editor of the U.S. Cycling Federation Rulebook and began systematically proposing amendments to the racing rules. Most of my proposed rule changes were accepted without controversy but my proposal in the early 1980s to require that strong helmets be used while racing ran into a brick wall because it constituted a strong break with tradition. I had based that proposal on my observation that nearly all serious injuries or deaths incurred in races were the direct result of head injuries.


While looking around for support I came across an article in a national cycling magazine advocating the same thing, which was written by a Dr. Robert Bond.  I recalled hearing that my former friend had gone off to Stanford and had become an M.D., so I wondered if he might be the author. It turned out that he was, so we joined forces and the next time I visited San Diego we and our wives got together and reminisced a bit but didnt discuss how Bobby and I had been separated.


We both continued to advocate adopting strong helmet requirements but my own bicycle club opposed me and booted me off their board of directors, then campaigned against my reelection to the national board of directors with the result that I lost my seat there for two years. As I departed I wrote a report refuting all known arguments against the strong helmet rule. By chance another cyclist died as a result of wearing an inadequate helmet at a velodrome near where the board held their next meeting. They finally saw the light and adopted a strong helmet rule.


Two years later I regained my seat on the board as the riders finally figured out that the strong helmet rule was a good thing. It then started spreading around the world and has since become standard in racing organizations almost everywhere, saving hundreds of lives and preventing thousands of serious head injuries. Im proud of that.


Recreational cyclists often follow the lead of racers in their selection of equipment and attire. In the U.S. the use of strong helmets has now become widely accepted and even enacted into laws in some places, at least for younger riders. However that trend has been less pronounced in Europe and other parts of the world so far.


For more on the prolonged helmet fight in the U.S. see The brain bucket bash.


Close this section

TypeScript Progressive Web App Quickstart

This is a bootstrap for a TypeScript Progressive Web App.

Service workers are auto generated. Note that you will need to click "Update on reload" in the Application tab of Chrome Dev Tools to get the latest code on each refresh.

BrowserSync is integrated to automatically detect changes and trigger browser reloads.

This is not the ideal setup for your application, but simply a bootstrap to get started quickly.

This was built on top of existing code from the Angular 2 Quickstart repo (

To get started, clone the repo and run npm start. Chokidar will detect changes and regenerate service workers using sw-precache.


Get it now if it's not already installed on your machine.

Verify that you are running at least node v4.x.x and npm 3.x.x by running node -v and npm -v in a terminal/console window. Older versions produce errors.

We recommend nvm for managing multiple versions of node and npm.

Create a new project based on the QuickStart

Clone this repo into new project folder (e.g., my-proj).

git clone  my-proj
cd my-proj

Discard the .git folder..

rm -rf .git  # OS/X (bash)
rd .git /S/Q # windows

Delete non-essential files (optional)

You can quickly delete the non-essential files that concern testing and QuickStart repository maintenance (including all git-related artifacts such as the .git folder and .gitignore!) by entering the following commands while in the project folder:

OS/X (bash)
xargs rm -rf < non-essential-files.osx.txt
rm src/app/*.spec*.ts
rm non-essential-files.osx.txt
for /f %i in (non-essential-files.txt) do del %i /F /S /Q
rd .git /s /q
rd e2e /s /q

Create a new git repo

You could start writing code now and throw it all away when you're done. If you'd rather preserve your work under source control, consider taking the following steps.

Initialize this project as a local git repo and make the first commit:

git init
git add .
git commit -m "Initial commit"

Recover the deleted .gitignore from the QuickStart repository if you lost it in the Delete non-essential files step.

Create a remote repository for this project on the service of your choice.

Grab its address (e.g.<my-org>/my-proj.git) and push the local repo to the remote.

git remote add origin <repo-address>
git push -u origin master

Install npm packages

See npm and nvm version notes above

Install the npm packages described in the package.json and verify that it works:

Doesn't work in Bash for Windows which does not support servers as of January, 2017.

The npm start command first compiles the application, then simultaneously re-compiles and runs the lite-server. Both the compiler and the server watch for file changes.

Shut it down manually with Ctrl-C.

You're ready to write your application.

npm scripts

We've captured many of the most useful commands in npm scripts defined in the package.json:

  • npm start - runs the compiler and a server at the same time, both in "watch mode".
  • npm run build - runs the TypeScript compiler once and generates a service worker
  • npm run build:w - runs the TypeScript compiler in watch mode; the process keeps running, awaiting changes to TypeScript files and re-compiling when it sees them. The service worker is automatically regenerated as each cacheable asset changes.
  • npm run serve - runs the lite-server, a light-weight, static file server, written and maintained by John Papa and Christopher Martin with excellent support for Angular apps that use routing.

Here are the test related scripts:

  • npm test - compiles, runs and watches the karma unit tests
  • npm run e2e - compiles and run protractor e2e tests, written in Typescript (*e2e-spec.ts)


The QuickStart documentation doesn't discuss testing. This repo adds both karma/jasmine unit test and protractor end-to-end testing support.

These tools are configured for specific conventions described below.

It is unwise and rarely possible to run the application, the unit tests, and the e2e tests at the same time. We recommend that you shut down one before starting another.

Unit Tests

TypeScript unit-tests are usually in the src/app folder. Their filenames must end in .spec.ts.

Look for the example src/app/app.component.spec.ts. Add more .spec.ts files as you wish; we configured karma to find them.

Run it with npm test

That command first compiles the application, then simultaneously re-compiles and runs the karma test-runner. Both the compiler and the karma watch for (different) file changes.

Shut it down manually with Ctrl-C.

Test-runner output appears in the terminal window. We can update our app and our tests in real-time, keeping a weather eye on the console for broken tests. Karma is occasionally confused and it is often necessary to shut down its browser or even shut the command down (Ctrl-C) and restart it. No worries; it's pretty quick.

End-to-end (E2E) Tests

E2E tests are in the e2e directory, side by side with the src folder. Their filenames must end in .e2e-spec.ts.

Look for the example e2e/app.e2e-spec.ts. Add more .e2e-spec.js files as you wish (although one usually suffices for small projects); we configured Protractor to find them.

Thereafter, run them with npm run e2e.

That command first compiles, then simultaneously starts the lite-server at localhost:8080 and launches Protractor.

The pass/fail test results appear at the bottom of the terminal window. A custom reporter (see protractor.config.js) generates a ./_test-output/protractor-results.txt file which is easier to read; this file is excluded from source control.

Shut it down manually with Ctrl-C.

Close this section

Show HN: Sorting Two Metric Tons of Lego

One of my uncles cursed me with the LEGO bug, when I was 6 he gave me his collection because he was going to university. My uncle and I are relatively close in age, my dad was the eldest of 8 children and he is the youngest. So, for many years I did nothing but play with lego, build all kinds of machinery and in general had a great time until I discovered electronics and computers.

So, my bricks went to my brother, who in turn gave them back to my children when they were old enough and so on. By the time we reached 2015 this had become a nice collection, but nothing you’d need machinery for to sort it.

That changed. After a trip to lego land in Denmark I noticed how even adults buy lego in vast quantities, and at prices that were considerably higher than what you might expect for what is essentially bulk ABS. Even second hand lego isn’t cheap at all, it is sold by the part on specialized websites, and by the set, the kilo or the tub on ebay.

After doing some minimal research I noticed that sets do roughly 40 euros / Kg and that bulk lego is about 10, rare parts and lego technic go for 100’s of euros per kg. So, there exists a cottage industry of people that buy lego in bulk, buy new sets and then part this all out or sort it (manually) into more desirable and thus more valuable groupings.

I figured this would be a fun thing to get in on and to build an automated sorter. Not thinking too hard I put in some bids on large lots of lego on the local ebay subsidiary and went to bed. The next morning I woke up to a rather large number of emails congratulating me on having won almost every bid (lesson 1: if you win almost all bids you are bidding too high). This was both good and bad. It was bad because it was probably too expensive and it was also bad because it was rather more than I expected. It was good because this provided enough motivation to overcome my natural inertia to actually go and build something.

And so, the adventure started. In the middle of picking up the lots of lego my van got stolen so we had to make do with an elderly espace, one lot was so large it took 3 trips to pick it all up. By the time it was done a regular garage was stacked top-to-bottom with crates and boxes of lego. Sorting this manually was never going to work, some trial bits were sorted and by my reckoning it would take several life times to get that all organized.

Computer skills to the rescue! A first proof of concept was built of - what else - lego. This was hacked together with some python code and a bunch of hardware to handle the parts. After playing around with that for a while it appeared there were several basic problems that needed to be solved, some obvious, some not so obvious. A small collection:

fake parts needed to be filtered out

There is a lot of fake lego out there. The problem is that fake lego is worth next to nothing and if a fake part is found in a lot it devalues that lot tremendously because you now have to check each and every part to make sure you don’t accidentally pass on fake lego to a customer.

discolored parts

Lego is often assembled as a set and then put on display. That’s nice, but if the display location is in the sun then the parts will slowly discolor over time. White becomes yellow, blue becomes greenish, red and yellow fade and so on. This would be fairly easy to detect if it wasn’t for the fact that lego has a lot of colors and some of the actual colors are quite close to the faded ones.

damaged parts

Not all Lego is equally strong, and some parts are so prone to breakage it is actually quite rare to find them in one piece. If you don’t want to pass on damaged parts to customers you need to have some way of identifying them and picking them out of the parts stream.

dirty parts

Most Lego that was bought was clean, but there were some lots that looked as if someone had been using them as growth substrate for interesting biological experiments. Or to house birds…

feeding lego reliably from a hopper is surprisingly hard

Lego is normally assembled by childrens hands, but a bit of gravity and some moving machine parts will sometimes do an excellent job of partially assembling a car or some other object. This tendency is especially pronounced when it comes to building bridges, and I’ve yet to find a hopper configuration wide and deep enough that a random assortment of Lego could not form a pretty sturdy bridge across the span.

The current incarnation uses a slow belt to move parts from the hopper onto a much faster belt that moves parts past the camera.

scanning parts

Scanning parts seems to be a trivial optical exercise, but there are all kinds of gotchas here. For instance, parts may be (much!) longer than what fits under the camera in one go, parts can have a color that is extremely close to the color of the background and you really need multiple views of the same part. This kept me busy for many weeks until I had a setup that actually worked.

parts classification

Once you can reliably feed your parts past the camera you have to make sense of what you’re looking at. There are 38000+ shapes and there are 100+ possible shades of color (you can roughly tell how old someone is by asking them what lego colors they remember from their youth). After messing around with carefully crafted feature detection, decision trees, bayesian classification and other tricks I’ve finally settled on training a neural net and using that to do the classification. It isn’t perfect but it is a lot easier than coding up features by hand, many lines of code, test cases and assorted maintenance headaches were replaced by a single classifier based on the VGG16 model but with some Lego specific tweaks and then trained on large numbers of images to get the error rate to something acceptable. The final result classifies a part in approximately 30 ms on a GTX1080ti Nvidia GPU. One epoch of training takes longer than I’m happy with but that only has to be done once.

distributing parts to the right bin

This also was an interesting problem, after some experimenting with servos and all kinds of mechanical pushers the final solution here was to simply put a little nozzle next to the transport belt and to measure very precisely how long it takes to move a part from the scan position to the location of the nozzles. A well placed bin then catches the part.

Building all this has been a ton of fun. As I wrote above the prototype was made from Lego, the current one is a hodge-podge of re-purposed industrial gear, copious quantities of crazy glue and a heavily modified home running trainer that provides the frame to attach all the bits and pieces to.

Note that this is by no means finished but it’s the first time that all the parts have come together and that it actually works well enough that you can push kilos of Lego through it without interruption. The hopper mechanism can still be improved a lot, there is an easy option to expand the size of the bins and there are still obvious improvements on the feeder. The whole thing runs very quiet, a large factor in that is that even though the system uses compressed air the compressor is not your regular hardware store machine but one that uses two freezer motors to very quietly fill up the reserve tank.

Here is a slow run tracking some parts so you can see how all the bits work together (it can run much faster):

A faster run, still slow enough that you can hopefully see what is going on:

Close this section

Archive that can be reconstructed with total loss of file system structure

A single file container/archive that can be reconstructed even after total loss of file system structures.


An SBX container exists both as a normal file in a mounted file system, and as a collection of recognizable blocks at a lower level.

SBX blocks have a size sub-multiple/equal to that of a sector, so they can survive any level of fragmentation. Each block have a minimal header that include a unique file identifier, block sequence number, checksum, version. Additional, non critical info/metadata are contained in block 0 (like name, file size, crypto-hash, other attributes, etc.).

If disaster strikes, recovery can be performed simply scanning a volume/image, reading sector sized slices and checking blocks signatures and then CRCs to detect valid SBX blocks. Then the blocks can be grouped by UIDs, sorted by sequence number and reassembled to form the original SeqBox containers.

It's Magic

It's also possible and entirely transparent to keep multiple copies of a container, in the same or different media, to increase the chances of recoverability. In case of corrupted blocks, all the good ones can be collected and reassembled from all available sources.

The UID can be anything, as long as is unique for the specific application. It could be random generated (probably the most common option), or a hash of the file content, or a simple sequence, etc.

Overhead is minimal: for SBX v1 is 16B/512B (+1 optional 512B block), or < 3.5%.

Demo tour

The two main tools are obviously the encoder & decoder:

  • SBXEnc: encode a file to a SBX container
  • SBXDec: decode SBX back to original file; can also show info on a container and tests for integrity against a crypto-hash

The other two are the recovery tools:

  • SBXScan: scan a set of files (raw images, or even block devices on Linux) to build a Sqlite db with the necessary recovery info
  • SBXReco: rebuild SBX files using data collected by SBXScan

There are in some case many parameters but the default are sensible so it's generally pretty simple.

Now to a practical example: let's see how 2 photos and their 2 SBX encoded versions go trough a fragmented floppy disk that have lost its FAT (and any other system part). We start with the 2 pictures, about 200KB and 330KB:

Castle Lake

We encode using SBXEnc, and then test the new file with SBXDec, to be sure all is OK:

C:\t>sbxenc Lake.jpg
hashing file 'Lake.jpg'...
SHA256 3cfc376b6362444d2d25ebedb19e7594000f2ce2bdbb521d98f6c59b5adebfdc
creating file 'Lake.jpg.sbx'...
SBX file size: 343040 - blocks: 670 - overhead: 3.4%

C:\t>sbxdec -t Lake.jpg.sbx
decoding 'Lake.jpg.sbx'...
metadata block found!
SBX decoding complete
SHA256 3cfc376b6362444d2d25ebedb19e7594000f2ce2bdbb521d98f6c59b5adebfdc
hash match!

Same for the other file. Now we put both the JPEG and the SBX files in a floppy disk image already about half full, that have gone trough various cycles of updating and deleting. As a result the data is laid out like this:

Disk Layout

Normal files (pictures included) are in green, and the two SBX in different shades of blue. Then with an hex editor with zap the first system sectors and the FAT (in red)! Time for recovery!

We start with the free (GPLV v2+) PhotoRec, which is the go-to tool for these kind of jobs. Parameters are set to "Paranoid : YES (Brute force enabled)" & "Keep corrupted files : Yes", to search the entire data area. As the files are fragmented, we know we can't expect miracles. The starting sector of the photos will be surely found, but as soon as the first contiguous fragment end, it's anyone guess.

PhotoRec results

As expected, something has been recovered. But the 2 files size are off (280K and 400KB). The very first parts of the photos are OK, but then they degrade quickly as other random blocks of data where mixed in. We have all seen JPEGs ending up like this:

Castle Lake

Other popular recovery tools lead to the same results. It's not anyone fault: it's just not possible to know how the various fragment are concatenated, without an index or some kind of list (there are approaches based on file type validators that can in at least some cases differentiate between spurious and valid blocks, but that's beside the point).

But with a SBX file it's a different story. Each one of its block can't be fragmented more, and contains all the needed data to be put in its proper place in sequence. So let's proceed with the recovery of the SBX files. To spice things up, the disk image file is run trough a scrambler, that swaps variable sized blocks of sectors around. The resulting layout is now this:


Pretty nightmarish! Now on to SBXScan to search for pieces of SBX files around, and SBXReco to get a report of the collected data:

C:\t\recovered\sbx>sbxscan \t\scrambled.IMA
creating 'sbxscan.db3' database...
scanning file/device '\t\scrambled.IMA' (1/1)...
100.0% blocks: 1087 - meta: 2 - files: 2 - 89.97MB/s
scan completed!

C:\t\recovered\sbx>sbxreco sbxscan.db3 -i
opening 'sbxscan.db3' recovery info database...

"UID", "filesize", "sbxname", "filename"
"2818b123c00b", 206292, "Castle.jpg.sbx", "Castle.jpg"
"76fe4a49ebf2", 331774, "Lake.jpg.sbx", "Lake.jpg"

The 2 SBX container have been found, with all the metadata. So the original filesizes are also known, along with the names of the SBX files and the original ones. At this point it would be possible to recover singles files or a group of them, by UID or names, but we opt to recover everything:

C:\t\recovered\sbx>sbxreco sbxscan.db3 --all
opening 'sbxscan.db3' recovery info database...
recovering SBX files...
UID 2818b123c00b (1/2)
  blocks: 417 - size: 213504 bytes
  to: 'Castle.jpg.sbx'
  100.0%   (missing blocks: 0)
UID 76fe4a49ebf2 (2/2)
  blocks: 670 - size: 343040 bytes
  to: 'Lake.jpg.sbx'
  100.0%   (missing blocks: 0)

all SBx files recovered with no errors!

All SBX files seems to have been recovered correctly. We start decoding:

C:\t\recovered\sbx>sbxdec Lake.jpg.sbx
decoding 'Lake.jpg.sbx'...
metadata block found!
creating file 'Lake.jpg'...
SBX decoding complete
SHA256 3cfc376b6362444d2d25ebedb19e7594000f2ce2bdbb521d98f6c59b5adebfdc
hash match!

And sure enough:

Castle Lake

N.B. Here's a 7-Zip archive with the 2 disk images used in the demo (542KB).

Possible / hypothetical / ideal uses cases

  • Last step of a backup. After creating a compressed archive of something, the archive could be SeqBox encoded to increase recovery chances in the event of some software/hardware issues that cause logic / file system's damages.
  • Exchange data between different systems. Regardless of the file system used, an SBX container can always be read/extracted.
  • Long term storage. Since each block is CRC tagged, and a crypto-hash of the original content is stored, bitrot can be easily detected. In addition, if multiple copies are stored, in the same or different media, the container can be correctly restored with high degree of probability even if all the copies are subject to some damages (in different blocks).
  • Encoding of photos on a SDCard. Loss of images on perfectly functioning SDCards are known occurrences in the photography world, for example when low on battery and maybe with a camera/firmware with suboptimal monitoring & management strategies. If the photo files are fragmented, recovery tools can usually help only to a point.
  • On-disk format for a File System. The trade-off in file size and performance (both should be fairly minimal anyway) could be interesting for some application. Maybe it could be a simple option (like compression in many FS). I plan to build a simple/toy FS with FUSE to test the concept, time permitting.
  • Easy file splitting. Probably less interesting, but a SeqBox container can also be splitted with no particular precautions aside from doing that on block size boundaries. Additionally, there's no need to use special naming conventions, numbering files, etc., as the SBX container can be reassembled exactly like when doing a recovery.
  • Data hiding. SeqBox containers (or even fragments of them) can be put inside other files (for example at the end of a JPEG, in the middle of a document, etc.), sprayed somewhere in the unused space, between partitions, and so on. Incidentally, that means that if you are in the digital forensics sector, now you have one more thing to check for! If a password is used, the entire SBX file is mangled to look pseudo-random, and SBXScan, SBXReco & SBXDec will not be able to recognize it unless the same password is provided.


SeqBox recoverability have been practically tested with a number of File Systems. The procedure involved using a Virtual Machine (or a full blown emulator) to format a small disk image with a certain FS, filling it with a number of small files, then deleting some of them randomly to free enough space to copy a series of SBX files. This way every SBX file results fragmented in a lot of smaller pieces. Then the image was quick-formatted, wipefs-ed and the VM shutdown. After that, from the host OS, recovery of the SBX files was attempted using SBXScan & SBXReco on the disk image.

Being written in Python 3, SeqBox tools are naturally multi-platform and have been tested successfully on various versions of Windows, on OS X & macOS, on some Linux distros either on x86 or ARM, on FreeBSD and on Android (via QPython).

Tech spec

Byte order: Big Endian

Common blocks header:

pos to pos size desc
0 2 3 Recoverable Block signature = 'SBx'
3 3 1 Version byte
4 5 2 CRC-16-CCITT of the rest of the block (Version is used as starting value)
6 11 6 file UID
12 15 4 Block sequence number

Block 0

pos to pos size desc
16 n var encoded metadata
n+1 blockend var padding (0x1a)

Blocks > 0 & < last:

pos to pos size desc
16 blockend var data

Blocks == last:

pos to pos size desc
16 n var data
n+1 blockend var padding (0x1a)


N.B. Current versions differs only by blocksize.

ver blocksize note
1 512 default
2 128
3 4096

Metadata encoding:

Bytes Field
3 ID
1 Len
n Data


ID Desc
FNM filename (utf-8)
SNM sbx filename (utf-8)
FSZ filesize (8 bytes)
FDT date & time (8 bytes, seconds since epoch)
SDT sbx date & time (8 bytes)
HSH crypto hash (SHA256, using Multihash protocol)
PID parent UID (not used at the moment)

(others IDs for file dates, attributes, etc. will be added...)

Final notes

The code was quickly hacked together in spare slices of time to verify the basic idea, so it's not optimized for speed and will benefit for some refactoring, in time. Still, the current block format is stable and some precautions have been taken to ensure that any encoded file could be correctly decoded. For example, the SHA256 hash that is stored as metadata is calculated before any other file operation. So, as long as a newly created SBX file is checked as OK with SBXDec, it should be OK. Also, SBXEnc and SBXDec by default don't overwrite files, and SBXReco uniquify the recovered ones. Finally, the file content is not altered in any way (except if a password is used), just re-framed.



If you need more info, want to get in touch, or donate: Marco Pontello

Bitcoin: 1Mark1tF6QGj112F5d3fQALGf41YfzXEK3


Close this section

The Invisible War for the Open Internet

There are a lot of scary things happening these days, but here’s what keeps me up late at night. A handful of corporations are turning our open internet into this:

These corporations want to lock down the internet and give us access to nothing more than a few walled gardens. They want to burn down the Library of Alexandria and replace it with a magazine rack.

Why? Because they’ll make more money that way.

This may sound like a conspiracy theory, but this process is moving forward at an alarming rate.

History is repeating itself.

So far, the story of the internet has followed the same tragic narrative that’s befallen other information technologies over the past 160 years:

  • the telegram
  • the telephone
  • cinema
  • radio
  • television

Each of these had roughly the same story arc:

  1. Inventors discovered the technology.
  2. Hobbyists pioneered the applications of that technology, and popularized it.
  3. Corporations took notice. They commercialized the technology, refined it, and scaled it.
  4. Once the corporations were powerful enough, they tricked the government into helping them lock the technology down. They installed themselves as “natural monopolies.”
  5. After a long period of stagnation, a new technology emerged to disrupt the old one. Sometimes this would dislodge the old monopoly. But sometimes it would only further solidify them.

This loop has repeated itself so many times that Tim Wu — the Harvard law professor who coined the term “Net Neutrality” — has a name for it: The Cycle.

“History shows a typical progression of information technologies, from somebody’s hobby to somebody’s industry; from jury-rigged contraption to slick production marvel; from a freely accessible channel to one strictly controlled by a single corporation or cartel — from open to closed system.” — Tim Wu

And right now, we’re in step 4 the open internet’s narrative. We’re surrounded by monopolies.

The problem is that we’ve been in step 4 for decades now. And there’s no step 5 in sight. The creative destruction that the Economist Joseph Schumpeter first observed in the early 1900s has yet to materialize.

The internet, it seems, is special. It’s the ultimate information technology — capable of supplanting the telegram, telephone, radio, cinema, television, and much more — and there’s no clear way to disrupt it.

But the war for the commanding heights of the internet is far from over. There are many players on this global chess board. Governments. Telecom monopolies. Internet giants like Google and Facebook. NGOs. Startups. Hackers. And — most importantly — you.

The war for the open internet is the defining issue of our time. It’s a scramble for control of the very fabric of human communication. And human communication is all that separates us from the utopia that thousands of generations of our ancestors slowly marched us toward — or the Orwellian, Huxleyan, Kafkaesque dystopia that a locked-down internet would make possible.

By the end of this article, you’ll understand what’s happening, the market forces that are driving this, and how you can help stop it. We’ll talk about the brazen monopolies who maneuver to lock down the internet, the scrappy idealists who fight to keep it open, and the vast majority of people who are completely oblivious to this battle for the future.

In Part 1, we’ll explore what the open internet is and delve into the history of the technological revolutions that preceded it.

In Part 2, we’ll talk about the atoms. The physical infrastructure of the internet. The internet backbone. Communication satellites. The “last mile” of copper and fiber optic cables that provide broadband internet.

In Part 3, we’ll talk about bits. The open, distributed nature of the internet and how it’s being cordoned off into walled gardens by some of the largest multinational corporations in the world.

In Part 4, we’ll explore the implications of all this for consumers and for startups. You’ll see how you can help save the open internet. I’ll share some practical steps you can take as a citizen of the internet to do your part and keep it open.

This is a long read. So grab a hot beverage and get ready to download a massive corpus of technology history into your brain.

Part 1: What is the open internet?

“Number 31” by Jackson Pollock. 1950. Household paints on canvas.

There’s only one word to describe the open internet: chaos.

The open internet is a cacophony of 3 billion voices screaming over one another. It’s a dusty, sprawling bazaar. And it’s messy. But it has produced some of the greatest art and industry of our time.

The open internet is a Miltonian marketplace of ideas, guided by a Smithian invisible hand of free-market capitalism.

The open internet is distributed. It’s owned in part by everyone and in whole by no one. It exists largely outside of the boundaries of governments. And it’s this way by design.

This reflects the wisdom of Vint Cerf, Bob Khan, J. C. R. Licklider, and all the wizards who stayed up late and pioneered the internet. They had seen the anti-capitalist, corporatists fate that befell the telegram, the telephone, the radio, and the TV. They wanted no part of that for their invention.

The open internet is a New Mexico Quilter’s Association. It’s a Jeremy Renner fan club. It’s a North Carolina poetry slam. It’s a Washington D.C. hackerspace. It’s a municipal website for Truckee, California. It’s a Babylon 5 fan fiction website.

The open internet is a general purpose tool where anyone can publish content, and anyone can then consume that content. It is a Cambrian Explosion of ideas and of execution.

Can these websites survive in a top-down, command-and-control closed internet? Will they pay for “shelf space” on a cable TV-like list of packages? Will they pay for a slice of attention in crowded walled gardens?

We’re all trapped in The Cycle

Here’s a brief history of the information technologies that came before the internet, and how quickly corporations and governments consolidated them.

Originally anyone could string up some cable, then start tapping out Morse Code messages to their friends. The telegram was a fun tool that had some practical applications, too. Local businesses emerged around it.

That changed in 1851 when Western Union strung up transcontinental lines and built relay stations between them.

If small telegraph companies wanted to be able to compete, they needed access to Western Union’s network. Soon, they were squeezed out entirely.

At one point Western Union was so powerful that it was able to single-handedly install a US President. If you grew up in America, you may have memorized this president’s name as a child: Rutherford B. Hayes.

Not only did Western Union back Hayes’ campaign financially, it also used its unique position as the information backbone for espionage purposes. It was able to read telegrams from Hayes’ political opponents and make sure Hayes was always one step ahead.

Western Union’s dominance — and monopoly pricing — would last for decades until Alexander Graham Bell disrupted its business with his newly-invented telephone.

How the telephone fell victim to The Cycle

After a period of party lines and local telephone companies, AT&T — backed by JP Morgan — built a network of long-distance lines throughout America.

In order for the customers of local phone companies to be able to call people in other cities, those companies had to pay AT&T for the privilege of using its long-distance network.

Theodore Vail — a benevolent monopolist if there ever was one — thought that full control of America’s phone systems was the best way to avoid messy, wasteful capitalistic competition. He argued that his way was better for consumers. And to be fair, it was. At least in the short run.

Vail was able to use AT&T’s monopoly profits to subsidize the development of rural phone lines. This helped him rapidly connect all of America and unify it under a single standardized system.

But the problem with benevolent monopolists is they don’t live forever. Sooner or later, they are replaced by second-generation CEOs, who often lack any of their predecessors’ idealism. They are only after one thing — the capitalist’s prerogative — maximizing shareholder value. That means making a profit, dispersing dividends, and beating quarterly earnings projections. Which means extracting as much money from customers as possible.

AT&T eventually squeezed out their competitors completely. And once AT&T’s monopoly became apparent, the US Government took action to regulate it. But AT&T was much smarter than its regulators, and jumped on an opportunity to become a state-sponsored “natural monopoly.”

AT&T would enjoy monopoly profits for decades before being broken up by the FCC in 1982.

But the “baby bells” wouldn’t stay divided for long. In 1997, they were able to start merging back together into a corporation even bigger than before the break-up.

The end result is one of the most powerful corporations on the planet — strong enough to expand its monopoly from the land-line telephone industry to the emerging wireless telecom industry.

AT&T functioned like a branch of government and had extensive research labs, but with one major exception — it could keep secret any inventions that it perceived as a threat to its core business.

Voicemail — and digital tape, which was later used as a critical data storage medium for computers — was actually invented within one of AT&T’s labs in 1934. But they buried it. It was only re-invented decades later.

Imagine how much progress the field of information technology could have made during that length of time with such a reliable and high-volume data storage medium at its disposal.

To give you some idea of how much just this one AT&T decision may have cost humanity, imagine that a corporation purposefully delayed the introduction of email by a decade. What would be the total impact on the productivity of society? How many trillions of dollars in lost economic activity would such an action cost us? This is the cautionary tale of what happens when you leave scientific research and development to private industry instead of public labs and universities.

You can still feel the legacy of AT&T’s monopoly when you call an older person from out of state. They will instinctively try to keep the call as short as possible, because they want to avoid the massive long distance fees historically associated with such calls, even though these no longer apply.

I thought this was just my grandmother, but it’s everyone’s grandmother. Entire generations have been traumatized by AT&T’s monopolistic pricing.

How cinema fell victim of The Cycle

Shortly after the invention of cinema, we had thousands of movie theaters around the US showing a wide variety of independently-produced films on all manner of topics. Anyone could produce a film, then screen it at their local theater.

That changed when Adolf Zukor founded Paramount Pictures. He pioneered the practice of “block booking.” If small independent theaters wanted to screen, say, the newest Mae West film, they would also need to purchase and screen a bunch of other lessor films.

This took away theater owners’ status as local tastemakers, and removed their ability to cater to their own local demographics. The result was the commoditization of movie theaters, and ultimately the rise of blockbuster cinema.

How radio fell victim to The Cycle

Shortly after Marconi — or Tesla — invented the radio, a massive hobbyist movement sprung up around it. There were thousands of local radio stations playing amateur programs.

In stepped David Sarnoff as the head of the Radio Corporation of America (RCA). He was perhaps the most Machiavellian CEO of the 20th century.

At the time, RCA was making parts for radio. Conventional thinking at the time was that RCA should focus on hardware, and getting as many radio stations running and as many radios into homes as possible. But Sarnoff realized that the real money was in content. He helped popularize the National Broadcast Corporation (NBC) and focused instead on making money through advertisements.

Then Sarnoff approached the Federal Radio Commission — now the Federal Communications Commission (FCC) — and convinced them that since the radio spectrum was a scarce commodity, they should carve it up and issue licenses.

Soon, NBC was available in every home, and the local hobbyist radio stations were squeezed off the air. RCA was now vertically integrated — from the parts in the radio stations, to the parts in consumer radios, to the content being broadcast itself.

Sarnoff had talked with the inventors of TV, and knew that it would eventually disrupt radio. But he had a plan. To claim the invention of television for himself.

How TV fell victim to The Cycle

TV is different from other forms of technology here, in that it didn’t enjoy a hobbyist stage. With the help of the FCC, Sarnoff and RCA immediately locked TV down. The result was several decades where Americans had just three channels to choose from — NBC, CBS, and ABC.

This was the height of mass culture — half of all Americans watching the same episode of I Love Lucy at the same time. The popularity of television — combined with the lack of diversity in programming caused by this monopoly — had social and political consequences that haunt us to this day.

Will the open internet fall victim to The Cycle?

We’ve gone through the invention step. The infrastructure came out of DARPA and the World Wide Web itself came out of CERN.

We’ve gone through the hobbyist step. Everyone now knows what the internet is, and some of the amazing things it’s capable of.

We’ve gone through the commercialization step. Monopolies have emerged, refined, and scaled the internet.

But the question remains: can we break with the tragic history that has befallen all prior information empires? Can this time be different?

Part 2: The War for Atoms

“IBM Cable Ball” by David Lan. 2007. Cables.
“Any sufficiently advanced technology is indistinguishable from magic.” — Arthur C. Clarke’s Third Law

As much as we may think of the internet as a placeless realm of pure abstractions, it has a physical structure. It’s not magic. And more people are waking up to this reality each day.

The internet is a series of copper and fiber optic cables that burrow through the ground and tunnel under oceans. We call this the Internet Backbone. Here’s what it looks like:

The internet is then further distributed through regional backbones. Here’s all the fiber that carries internet data around the United States. Red squares represent the junctions between “long haul” fibers.

Image credit: InterTubes: A Study of the US Long-haul Fiber-optic

The invisible workhorses of the internet: backbone providers

Six major companies control the backbone, and they “hand off” traffic from one another without any money exchanging hands:

  • Level 3 Communications
  • Telia Carrier
  • NTT
  • Cogent
  • GTT
  • Tata Communications.

Within the US, the backbone is mostly controlled by old long distance carriers, including Verizon and AT&T — who also control a two thirds of America’s $200 billion wireless industry.

These companies “peer” traffic through backbone connections controlled by other companies, or pay each other through “transit agreements.”

Despite the involvement of these huge telecoms, the internet backbone represents a fairly healthy market. About 40% of the internet’s backbone is controlled by smaller networks you’ve never heard of.

The mafia of the internet: the ISPs

The broadband internet market, on the other hand, isn’t healthy at all. This is the “last mile” of cables that plug into the internet backbone. And it’s full of ugly tollbooths, guarded by thick benches of lawyers and lobbyists.

This broadband internet market is controlled by just three extremely powerful — and widely hated — internet service providers (ISPs):

Another form of ISPs are the wireless providers:

  • AT&T
  • Verizon (formerly part of AT&T)

These two providers control 2/3rd of the wireless market. If you have a mobile phone, there’s a good chance you pay one of these companies every month for your data plan.

These ISPs control millions of miles of copper cables that they buried in the ground back in the 1970s, and satellites they shot up into orbit in the 1990s. They constantly break the law, tie up regulators in lengthy court battles, and make it practically impossible for anyone — even Google — to enter their markets.

The ISPs do all this for one reason and one reason alone: so they can avoid free market competition — and the expensive technology upgrades it would require — while they continue raking in their monopoly rents from the 2/3 of Americans who only have one choice in their neighborhood for broadband internet.

For the past two years, the public had a weapon against these ISPs. It’s not one that can mortally wound them , but it has helped beat back their monopolistic tendencies. It’s called Net Neutrality.

How Net Neutrality works

The story of ISPs basically comes down to this: They used to make a ton of money off of cable packages. But people discovered that once they had the internet, they didn’t care about cable TV any more — they just wanted data plans and so they could watch YouTube, Netflix, or whatever shows they wanted — and they could also consume a lot of non-video content, too.

The ISPs don’t make nearly as much selling you a data plan as they used to make selling you a cable plan, though. So their goal is to return to the “good old days” by locking down the internet into “channels” and “bundles” then forcing you to buy those.

How do we prevent this? The good news is that we already have. In 2015, the FCC passed a law that regulated ISPs as utilities. This is based on the principle of “Net Neutrality” which basically states that all information passing through a network should be treated equally.

As part of its 2015 decision on Net Neutrality, the FCC asked for public comment on this topic. 3 million Americans wrote to the FCC. Less than 1% of those people were opposed to Net Neutrality.

After a hard fought battle against telecoms, we convinced the FCC to enshrine Net Neutrality into law.

The FCC’s Title II regulation created three “bright lines” that prevent ISPs from doing the following:

  1. Blocking content from websites
  2. Slowing down content from websites
  3. Accepting money from websites to speed up their content

These rules made it so that no matter how rich and powerful a corporation is — and Apple and Google are the biggest corporations on Earth, and Microsoft and Facebook aren’t far behind — they can’t buy priority access to the internet.

Everyone has to compete on a level playing field. These tech conglomerates have to compete with the scrappy startups, the mom-and-pop businesses, and even independent bloggers who are running WordPress on their own domain.

Nobody is above Net Neutrality. It’s as simple a tool as possible for protecting the capitalist free market internet from monopolies who would otherwise abuse their power.

Now ISPs are treated like a utility. How are the packets being routed through a network different from the water being piped through the ground, or the electricity flowing through a power grid?

The water company shouldn’t care whether you’re turning on a tap to wash dishes or to take a shower.

The power company shouldn’t care whether you’re plugging in a TV or a toaster.

The ISPs shouldn’t care what data you want or what you use it for.

The reason ISPs want to get rid of Net Neutrality is simple: if we stop treating them like the utility that they are, they can find ways to charge a lot more money.

Here’s the former CEO of AT&T laying out his evil plan:

“Now what they would like to do is use my pipes free, but I ain’t going to let them do that because we have spent this capital and we have to have a return on it. So there’s going to have to be some mechanism for these people who use these pipes to pay for the portion they’re using. Why should they be allowed to use my pipes? The Internet can’t be free in that sense, because we and the cable companies have made an investment and for a Google or Yahoo! or Vonage or anybody to expect to use these pipes [for] free is nuts!” — Edward Whitacre, AT&T CEO

What he should certainly realize is that everyone is already paying for internet access. You’re paying to be able to access this article. I’m paying to push this article up onto the internet. This website is paying to send the traffic from its servers over to your computer.

We have all already paid to use these ISP’s last mile of cables. No one is using these pipes for free.

But the ISPs see an opportunity to double dip. They want to charge for bandwidth, and also charge websites what the Mafia calls “protection money.” They essentially want to be able to say to website owners: “Those are some lovely data packets you’ve got there. It sure would be a shame if they got lost on their way to your users.”

Of course, most of the open internet couldn’t afford to pay this “protection money” to ISPs, so the ISPs would block traffic to their websites, cutting consumers off from most of the open internet. But the ISPs wouldn’t need to block these websites. All the ISPs would need to do is introduce a slight latency.

Both Google and Microsoft have done research that shows that if you slow down a website by even 250 milliseconds — about how long it takes to blink your eyes — most people will abandon that website.

That’s right — speed isn’t a feature, it’s a basic prerequisite for attracting an audience. We humans are extremely impatient and becoming more so with each passing year.

This means that in practice, if an ISP artificially slows down a website, it’s practically as damaging as blocking the site entirely. Both of these acts result in the same outcome — a severe loss of traffic.

Traffic is the lifeblood of websites. Without traffic, merchandise doesn’t get sold. Services don’t get subscribed to. Donations don’t get made.

Without traffic, the open web dies — whether ISPs block it or not.

The ISPs have launched an all-out assault on Net Neutrality

With January’s change in US administration and the election of our 45th president, the FCC has changed as well.

The FCC Chairman Ajit Pai — a former Verizon lawyer — is now in control of the only regulator that the ISPs answer to. And here’s a direct quote from him:

“We need to fire up the weed whacker and remove those rules that are holding back investment, innovation and job creation.” — FCC Chairman Ajit Pai

The ISPs won’t reinvest their “protection money” in infrastructure. They already have incredible monopoly profits. Here’s their net income (after-tax profits) from 2016:

  • AT&T: $16 billion
  • Verizon: $13 billion
  • Comcast $8 billion
  • Charter $8 billion

They have plenty of profit they could claw back into improving infrastructure. They’re choosing instead to disperse this money to shareholders.

In just two months, Chairman Pai has already done incredible damage to Net Neutrality. He dropped Zero Rating lawsuits against four monopolies who were in clear violation of Net Neutrality law. Now Comcast and AT&T can continue to stream their own video services without them counting toward customers’ data caps, and there’s nothing the FCC will do about it.

Former FCC Chairman Tom Wheeler did his best to reach out to Chairman Pai and convince him of the virtues of Net Neutrality. The two were scheduled to meet once every two weeks during Wheeler’s last 18 months in office. But Pai cancelled every single one of these meetings.

“You have to have open networks — permissionless innovation. Period. End of discussion. They’re crucial to the future.” — Former FCC Chairman Tom Wheeler

Part 3: The War for Bits

“Sweet as One” by Craig and Karl. 2016. Candy.

What does a post Net Neutrality internet look like? Look no further than the Apple App store.

There are two million apps in the app store, which shared a total of $28 billion in 2016. Apple takes a 30% commission on every sale, and made $8.4 billion from the app store alone.

Most of the remaining $20 billion goes to just a small handful of mobile gaming companies:

Most iPhone users download zero apps per month.

The minority who do bother to download new apps don’t end up downloading very many.

And all 8 of the top apps in the app store are owned by just two corporations: Facebook and Google.

A vast majority of the remaining 2 million apps get very little traffic — and even less money.

The Apple App Store isn’t a level playing field. It doesn’t resemble the open internet it was built on top of. Instead, it’s an example of a walled garden.

Walled gardens look beautiful. They’re home to the most popular flora. But make no mistake, you won’t be able to venture very far in any one direction without encountering a wall.

And every walled garden has a gatekeeper, who uproots plants that look like weeds. If you want to plant something in a walled garden, you have to get approval from that gatekeeper. And Apple is one of the most aggressive gatekeepers of all. It keeps out apps that compete with its own interests, and censors apps that don’t mesh with its corporate worldview.

A brief history of walled gardens

First there was the original walled garden of the internet, AOL.

20 years later, AOL still has 2 million users paying them $20/month. There’s a lot of money to be made in building walled gardens and trapping users in them.

Then came Yahoo, which wasn’t a walled garden by design, but became one anyway because people were so new to the internet.

In the late 90s, startups raised money specifically so they could buy banner ads on Yahoo. It was the best way they could reach prospective users.

But Yahoo was a candle in the sun compared to the ultimate walled garden: Facebook.

A quarter of the people on Earth use Facebook for an average of 50 minutes each day.

And those 50 million people connected to that Mark Zuckerberg is bragging about? Those are people from extremely poor countries who were given a choice: they could either pay for the open internet or just get Facebook for free. They chose Facebook.

The insidiously-named was famously rejected in India — among other countries — where activists were able to raise awareness about all the things Indians would give up by accepting Facebook instead of the open internet.

The zero in un-ironically stands for Zero Rating, an anti-Net Neutrality practice that’s illegal in most western countries.

Mark Zuckerberg may mean well, but he’s rapidly destroying the open internet. In his ravenous quest to expand Facebook’s market share, he’s even gone so far as to build a sophisticated censorship tool so that Facebook can appease the governments of countries where it’s currently blocked, like China.

And Facebook is just one of several internet corporations who stand to profit from these sort of closed-source, closed-data walled garden platforms.

Here are the 10 largest corporations in the world by market capitalization:

  1. Apple Inc
  2. Alphabet (Google)
  3. Microsoft
  4. Exxon Mobil
  5. Johnson & Johnson
  6. General Electric
  8. Facebook
  9. Wells Fargo
  10. AT&T

All of them are American-based multinationals. 6 out of 10 of them are internet companies, and one of them is an ISP.

Once you look past the last gasp of the banks and the oil companies, it becomes clear that these internet companies are the new order. They control information. They control the conversation. They control politics. Facebook won the new president the election — even the president and his advisors acknowledge this.

So what makes you think they won’t come to control the very internet they dominate?

Even as the costs of launching a website fall, the costs of reaching an audience continue to rise.

Facebook and Google account for 85% of all new dollars spent on online advertising. Everyone else — newspapers, blogs, video networks — is fighting for crumbs — the 15% that fell from Facebook’s and Google’s mouths.

Half of all internet traffic now comes from just 30 websites. The remaining half is thinly spread across the 60 trillion unique webpages currently indexed by Google.

If you’re familiar with the concept of a long tail distribution, you’ll recognize this phenomenon as an extremely fat head with an extremely long, skinny tail.

We blindly trust tech founders to be benevolent

You may think that the Mark Zuckerbergs and the Larry Pages of the world would know better than to abuse their power. But such scandals have happened in the past.

Reddit is one of the most popular websites on the internet. One of its founders recently put the company’s reputation in jeopardy. He admitted that he had modified users’ comments in Reddit’s database — essentially putting words in the mouths of people who were critical of him.

We are not only placing faith in the temperament of the elite handful of tech company founders. We’re also trusting that other actors — who ultimately take over these organizations — will be benevolent. Even when we know that their shareholders — or governments — can force them to be malevolent and do things that go against their users’ interests.

However you may feel about Mark Zuckerberg and his intentions, know this: Just like the “benevolent monopolist” Theodore Vail, who championed rural access to AT&T in the early 20th century, Mark Zuckerberg will one day retire. And the person who takes over Facebook will not be nearly as forward thinking as he is. Most likely, it will be some finance guy or sales guy who will sell Facebook users — and their Exabytes of data — down the river.

By destroying Net Neutrality, the ISP monopolies are herding us all into walled gardens

If we lose net neutrality, websites that once freely operated on the open internet will face three choices:

  1. pay ISPs so that their customers can access their website
  2. don’t pay ISPs, and plummet into obscurity
  3. become part of a walled garden that is paying ISPs on their behalf

This last option will be the most appealing for most small businesses. They will choose the free option. And in doing so, they’ll hand over to the walled gardens some amount of control over their own websites.

A Google or a Facebook will step in to help ensure that your customers are able to access your business’s website. These walled gardens will pay ISPs on your behalf, and help serve your content on their own domains. But in return, the walled garden could:

  • inject ads into your website (probably ads for your competitors)
  • capture your data and sell it (probably to your competitors)
  • redirect your customers to the websites of competitors who are willing to pay for your audience

Just like with Google ads or Facebook ads, the internet will become a race to see who can pay walled gardens the most money so they can gain access to customers. And most of this will be completely invisible to consumers.

There are precedents for all of this.

Facebook convinced millions of businesses to setup Facebook pages. The companies then spent their own money publicizing their Facebook pages and getting their customers to “like” their pages. Then Facebook pulled a bait-and-switch, and made it so these businesses would have to advertise through Facebook to reach their own customers who’d previously liked their pages.

And here’s what happens when a small nonprofit like freeCodeCamp refuses to pay for Google ads:

Companies with lots of money like this one — which is a subsidiary of Kaplan, one of the largest for-profit education conglomerates on earth — can pay money to Google so they can intercept our users.

And these ads will gradually look less and less like ads. Here’s how Google ads have changed over time to look more and more like normal search results:

Image credit

Now that tiny, green-bordered box with the word “ad” in it is all that distinguishes an advertisement from a legitimate search result. It is perhaps unsurprising that 55% of Google users don’t even recognize the fact that these are ads.

Eventually walled gardens may converge on something similar to Baidu, China’s largest search engine, which for a long time wasn’t labelling ads at all.

Baidu got into trouble last year after a college student used their search engine to seek treatment for a commonly treatable form of cancer. The student went to a hospital he found at the top of Baidu’s search results.

What the student didn’t know was that that hospital had paid Baidu’s money to be put at the top of the search results, and that this was in fact an advertisement. But Baidu had deliberately obscured this fact from their users so they could charge more for the ad.

The hospital proceeded to recommend an expensive and unproven drug instead of the standard — and far cheaper — treatment of surgery and chemotherapy.

After exhausting his family’s savings of $30,o00 on the ineffective treatment, the 21-year-old student wrote one final essay about his situation and how Baidu had lead him right into the hands of fraudsters. Then he died.

This is just a glimpse into the human toll that these walled gardens can inflict upon society. In a walled garden environment where only those who pay money get seen, consumers will face more misinformation, more fraud, and more needless suffering.

Instead of the equalizing force that was the open internet, the rich will get richer and the poor will get poorer. The internet’s promise of economic democratization will fall by the wayside, and we’ll enter yet another age of peasants living under feudal lords.

In the future, our internet could become as locked-down as China’s

China has the most sophisticated censorship tools in the world. So much so that other authoritarian regimes license the use of these tools to control their own populations.

1.4 billion Chinese people are locked down in a closed internet, behind the Great Firewall of China.

The anti-Net Neutrality agenda that the ISPs are pursuing would require them to use a technique called Deep Packet Inspection. Without looking inside the contents of every packet, it is impossible for ISPs to decide which packets they want to selectively slow down.

This means that in addition to sending packets of data through their networks, ISPs would actually have to look inside each of these packets — and would quite likely record the contents of these packets. It would be expensive, but storing major chunks of the Zettabyte of information the internet generates each year is within the budgets of large corporations and governments.

There’s a precedent for this, too. AT&T illegally monitored all of its traffic for years.

Monitoring internet traffic at this level of detail would make pervasive censorship possible. This is one of the techniques China uses to re-write its history. And it works. Despite the advances in information technology, to this day many Chinese still don’t know that the Tiananmen Massacre happened. And when they do learn of it, it’s ancient history — sapped of most of its perceived relevance.

“Ideas are more powerful than guns. We would not let our enemies have guns, why should we let them have ideas.” — Joseph Stalin

If the ISPs succeed and the open internet falls, corporations and governments would have a mandate to censor the most powerful communication tool in human history — the internet — in its entirety.

Part 4: Who controls the information? Who controls the future?

“Ascension” by Nathan Sawaya. 2014. Legos.

Whether these corporations are aggregating power through regulatory capture or by amassing exabytes of your data, they are steadily becoming more powerful. They are using their growing cashflow to buy up competitors.

This isn’t capitalism — it’s corporatism. Capitalism is messy. It’s wasteful. But it’s much healthier in the long run for society as a whole than central planning and government trying to pick the winners.

Capitalism allows for small businesses to enter and actually stand a chance. Corporatism makes it impossible.

If you’ve read this far, I hope you understand the gravity of this situation. This is not speculative. This is really happening. There are historical precedents. There are present-day examples.

If you do nothing, we will lose the war for the open internet. The greatest tool for communication and creativity in human history will fall into the hands of a few powerful corporations and governments.

Without your actions, corporations will continue to lock down the internet in ways that benefit them — not the public.

The good news is that our great grandparents reined in similar monopolies. At the beginning of the 20th century, Americans faced abusive oil, railroad, and meat industry monopolies. We prevailed over them by raising awareness through brave journalism, and compelling the government to act.

Today, our most urgent task at hand is stopping FCC Chairman Ajit Pai from disassembling Net Neutrality.

Help us fight this war. Here’s what I’m asking you to do:

  1. If you can afford to, donate to nonprofits who are fighting for the open internet: Free Press, the ACLU, the Electronic Frontier Foundation, and Public Knowledge.
  2. Educate yourself about the importance of the open internet. Read Tim Wu’s “The Master Switch: The Rise and Fall of Information Empires.” It is by far the best book on this topic.
  3. Contact your representatives and ask them what they’re doing to defend Net Neutrality.
  4. Share this article with your friends and family. I realize the irony of asking you to use walled gardens to spread the word, but this late in the game, these are the best tools available. Share this article on Facebook or tweet this article.

Only we, the public, can end The Cycle of closed systems. Only we can save the open internet.

Thank you for reading this, and for caring about the fate of our open internet.

Close this section

Award-Winning Nautilus Enters Rough Waters

In many respects, these are heady days for Nautilus, the highbrow science and culture magazine launched precisely four years ago this weekend. Traffic to its original website is setting records, its print edition has been well-received, and in its short existence, the magazine has earned numerous awards for its design and its journalism, including two National Magazine Awards — arguably, the industry’s highest honor.


More recently, it drew laurels for an essay by Cormac McCarthy — the first piece of nonfiction that the legendary novelist has ever published — and the magazine’s publisher has suggested that a major new partnership with a science organization is in the works.

For all the good news and accolades, however, murmurings within the science writing community suggest that not all is well at Nautilus. Rumors of delayed or entirely absent payments to the magazine’s fleet of freelance contributors have reached a crescendo, as have complaints that editorial staff continue to solicit work knowing that the publication may not be able to make good on promised fees.

One Nautilus freelancer, who asked to remain anonymous because thousands of dollars in fees are still pending, received a note a few months ago directly from the magazine’s publisher and editorial director, John Steele, offering assurances that the funds — which were for a feature that the magazine published last year — would be on their way by the end of January. That deadline came and went without payment, the freelancer said, and follow-up emails to Nautilus have not changed things.

“It’s sad,” the writer said, “because we need science publications that pay decently — if they actually pay.”

Linda Marsa, a Los Angeles-based journalist, had more success. Marsa wrote a piece last fall, submitted an invoice, and didn’t get paid on time. On Facebook, she learned that other writers were experiencing problems too. Frustrated, Marsa sent what she describes as a “stern” email to her editor, as well as the finance manager, the publisher, and a member of the Nautilus board.

That worked, and her payment arrived roughly a month overdue. When I called Marsa, she was surprised and upset to learn that the problems were continuing. Describing her editor at the magazine as “a gem,” Marsa said she’s nonetheless disillusioned and does not plan to write for the magazine again. “I’m really sad I’m not going to get to work with him again,” she said of her editor, “but this isn’t right.”

The publication’s exquisitely designed web site and print edition don’t come cheaply, and staff have taken pay cuts.

In a phone call with Undark, Steele, the Nautilus publisher, acknowledged that the magazine “has been running on fumes for the past six months,” and that it was behind on payment for many of its writers and illustrators. “We feel horrible about the way the situation has dragged on so far,” he said. “It tears me apart that I can’t fulfill these commitments that I’ve made to these guys, when they’ve given us such fantastic work. But I say, ‘Look, you’re going to get it. It’s going to happen. Just bear with me a little while longer.’”

Steele, a former television journalist, started Nautilus in 2012 with a two-year, $5 million grant from the John Templeton Foundation, a Pennsylvania-based philanthropy that describes itself as targeting the world’s “big questions” in science, religion, and philosophy. That money, supplemented with an additional $2.1 million from Templeton, was the main funding source for Nautilus in the run-up to publication, and during its first two years in print. But the Templeton Foundation typically reduces support for startup ventures after the first three years, and, accordingly, it has dialed back funding for Nautilus, although it gave the magazine an additional $1.25 million in 2015, and a little more than $1.2 million last year.

“It’s simply a matter of our desire to step back and not be a sustaining funder for the indefinite future, but rather allow [Nautilus] to grow and flourish,” said Christopher Levenick, the director of public engagement for the Templeton Foundation. He said he was unaware that Nautilus has been experiencing cash flow problems and failing to pay contributors.

Steele’s continued confidence that the magazine’s money problems will be short-lived is rooted in what he describes as the publication’s pending absorption by the American Association for the Advancement of Science — the world’s largest general scientific organization. The talks with AAAS began in December, Steele told me, after months of hunting for a new institutional home. The publication has also lined up funders who would step in and add additional resources once the deal with AAAS is complete. In principle, Nautilus would become a part of AAAS, and the funders will help put Nautilus in the black until it becomes profitable on its own, Steele added — though he declined to name the additional funders because the details have not been finalized.

Steele said he thought the arrangement with AAAS would proceed quickly, which is why he felt comfortable commissioning new work and telling freelancers that their money was on the way. But that didn’t happen. The process of coordinating among the funders and AAAS has moved slowly, and Steele told me that the magazine stopped commissioning new articles in late February or early March, once it became clear that they would not finalize the AAAS deal and have new funding as soon as he had anticipated.

Tiffany Lohwater, a spokeswoman at AAAS, did not respond to Undark’s request to discuss Nautilus over the phone, saying only in an email message that “Nautilus approached AAAS to inquire about the possibility of an affiliation, and AAAS is in the process of exploring that.” When pressed, she added: “AAAS is very much in the exploratory stage of any potential affiliation. There is no arrangement.”

That disconnect suggests, at the very least, that the lean times will continue — both for Nautilus, and for its unpaid contributors. Over the last several months, the magazine has gradually shrunk the size of its budget, and all staff have taken pay reductions. Today it earns revenue through individual donations and grants from other foundations; from subscriptions to its digital and print products (the latter is currently published in partnership with MIT Press); and, more recently, from branded, foundation-sponsored verticals, called channels, that are each devoted to a specific theme.

Steele says he intends to begin offering more writers partial payments beginning next month, drawing on revenue from Nautilus subscribers and sponsors. Asked whether Nautilus pays writers interest on their overdue fees, Steele said that he does so “on an individual basis” and when people specifically ask for additional compensation.

That might be welcome news for some writers, for whom even a single delayed payment can sometimes be financially disastrous. One contributor, who did not want to be named in order to preserve a working relationship with the magazine, told me that delayed payments from Nautilus had made it hard to pay the bills.

“I started pitching other places,” the freelancer said. “And I started dog walking.”

Close this section

Luxury Music Festival Turns Out to Be Half-Built Scene of Chaos

Fyre Festival attendees walk around the event site on Thursday night in Grand Exuma, Bahamas. Festivalgoers had paid thousands of dollars for villas and lodges on the beach, but instead found a chaotic tent city. Miles Braun hide caption

toggle caption
Miles Braun

Fyre Festival attendees walk around the event site on Thursday night in Grand Exuma, Bahamas. Festivalgoers had paid thousands of dollars for villas and lodges on the beach, but instead found a chaotic tent city.

Miles Braun

Perhaps you're a person who buys festival wear but finds Coachella too plebian. Perhaps you find other music festivals off-putting because you can't bring your own yacht. Or maybe you just think it sounds awesome to hang out on an island in the Bahamas and you have a few thousand dollars to blow.

In that case the Fyre Festival was supposed to be the event – nay, cultural moment! – for you.

(And yes, that's FYRE, not fire, because it was going to be LIT. And also because Fyre Media Inc. is the name of rapper Ja Rule's talent-booking company, which organized the luxstravaganza, or #dumpsterfyre, as you'll soon see.)

In a promo video posted in January full of frolicking models, the Fyre Festival promised (in seemingly random order) "the best in food, art, music and adventure / once owned by Pablo Escobar / on the boundaries of the impossible / Fyre is an experience and festival / A quest / to push beyond those boundaries."

Thursday was to be the first day of the two-weekend event.

"You'll be flown roundtrip on a custom, VIP configured Boeing 737 aircraft between Miami International Airport and Exuma International Airport on Great Exuma," said the festival's website. "Guests will be staying in modern, eco-friendly, geodesic domes. ... Unplug from the everyday and ignite your flame in the Exumas."

The acts scheduled to perform included Major Lazer and Blink-182, as well as a DJ "who specializes in producing '70s and '80s rock remixes for clients that include Middle Eastern and European royalty," as The Wall Street Journal reported.

The organizers built buzz by having celebrities tweet and Instagram about the festival. In a now-deleted tweet from March 27, Ja Rule wrote, "This is where the cool kids will be April 27-30 May 5-8!!! #fyrefestival #fyre."

Ticket prices were steep – but of course they were, since it was going to be "the cultural event of the decade." The Los Angeles Times reported in January that passes, which included accommodations and chartered flights from Miami, started at $1,595 and stretched to $399,995, which included dinner with a performer. (Though prices varied widely according to accommodation, and attendees interviewed by NPR said tickets could be had for as little has $900.)

But there were signs of trouble in paradise.

According to Journal's article from April 2, the organizers missed a series of payments to performers, though it had begun making progress in paying them. The newspaper noted that festivals typically lose money in their early years, and face high upfront costs.

No matter. Just days before the festival was to begin, @fyrefestival was shaking the sand out of its hair and Instagramming without concern.

But there was to be no dancing on the beach.

Yesterday, ticketholders began arriving at the festival, and found the site in disarray. Instead of the promised (and paid for) villas and lodges, festivalgoers found instead row after row of the same white tents.

"You just get dropped off on this island, and you're just standing there," said Seth Crossno, who had flown to the Bahamas with three other friends from Raleigh. "It looked like a disaster relief area." He said that cars and trucks were driving around, and that shipping containers littered the area.

Crossno said a man stood on a table shouting out directives.

"They had no way to communicate with anybody," said Crossno. "I don't know why no one went on the main stage and got on the microphone to get the crowd's attention and tell people what was going on. Just total incompetence." He added that house music blared in the background.

He estimated that a thousand people were there, and more kept arriving.

Crossno arrived at the festival around 5:30 p.m. Thursday night, and within a couple of hours, he and his friends were trying to get off the island. "It was a huge mess from the start," he said. Together, Crossno said he and his friends paid about $12,000 to attend.

Janan Buisier, who traveled from Dallas, said she spent more than $1,300 to go to the event. At first, she and her friends wanted to make the best of a bad situation, she said, "but then we saw how it started getting dangerous." She said the site didn't have enough security, lights, or food.

"You were promised chargers for your phone — did not get that," Busier said. "You were promised food — we were like starving. And you were promised safety, you were promised to be taken care of, you were promised an experience of a lifetime. And yes, it was quite the experience, but not in a positive way."

The view from Fyre Festival. Miles Braun hide caption

toggle caption
Miles Braun

The view from Fyre Festival.

Miles Braun

The festival started arranging flights to get people back to Miami, including ones that Crossno and Buisier were able to get on.

The Bahamas Ministry of Tourism tweeted out a statement Friday morning, saying it was "extremely disappointed in the way the events unfolded yesterday" and said that although it was not an official sponsor of the festival, it had lent its support when asked. "The event organizers assured is that all measures were taken to ensure a safe and successful event but clearly they did not have the capacity to execute an event of this scale."

It added that it hopes festival visitors would consider returning to the Bahamas in the future "to truly experience all of our beauty."

Not long after, the Fyre Festival website posted its own statement:

Fyre Festival set out to provide a once-in-a-lifetime musical experience on the Islands of the Exumas.

Due to circumstances out of our control, the physical infrastructure was not in place on time and we are unable to fulfill on that vision safely and enjoyably for our guests. At this time, we are working tirelessly to get flights scheduled and get everyone off of Great Exuma and home safely as quickly as we can. We ask that guests currently on-island do not make their own arrangements to get to the airport as we are coordinating those plans. We are working to place everyone on complimentary charters back to Miami today; this process has commenced and the safety and comfort of our guests is our top priority.

The festival is being postponed until we can further assess if and when we are able to create the high- quality experience we envisioned.

We ask for everyone's patience and cooperation during this difficult time as we work as quickly and safely as we can to remedy this unforeseeable situation. We will continue to provide regular updates via email to our guests and via our official social media channels as they become available.

-The Fyre Festival Team

Whether the circumstances were truly out of the Fyre Festival's control will probably be a matter for the lawyers. Crossno, who on Friday morning was back in Miami after a very strange trip, said that if he doesn't get his money back, he will sue.

But first, he has a warning for those who bought tickets for next weekend: "Do not go to this festival," he said. "It's a scam. It's not real."

All Things Considered producer Greg Dixon contributed to this report.

Close this section

China Doesn't Understand the Concept of American Chinese Food (2014)

This article originally appeared on MUNCHIES in May 2014. Fortune Cookie closed in January 2016.

If you're a Westerner, even if the closest you've got to Asian culture is stumbling across the Great Wall on Google Earth, you know that Chinese people don't crack open fortune cookies after every meal. And as a Brit living in Shanghai since a year ago, I can confirm that rather than sweet and sour chicken, most Chinese people prefer a nice pile of crispy chicken feet.

In fact, as the wonderfully named former New York Times reporter Jennifer 8. Lee pointed out in her 2008 TED Talk, most Chinese people don't even know what chop suey actually is. Since Chinese food first began being served in the USA in the 19th century, it has had generations to evolve and suit US tastes, so much so that it's completely disconnected to traditional dishes served in China, both now and then.

"In China they like bones, but we had the staff spend hours deboning the chicken," says Fung. "They were saying, 'Why are we doing this?'

Given that most Chinese people wouldn't recognize a plate of sticky orange chicken if it was splattered in their face, it seems an odd move to open an eatery almost exclusively serving American-style Chinese food in the middle of Shanghai. But that's what New Yorker Fung Lam and California-born Dave Rossi have done in the shape of Fortune Cookie, which opened ten months ago.

Photo by Jamie Fullerton

Above, Fung (left) and Dave (right). All photos by the author.

Having failed in their bid to launch a salad-based venue in Shanghai two years ago, the pair, who had quit white collar jobs and moved to China to try to launch a venue together, were craving American-Chinese comfort food and couldn't find it in China.

"When somebody feels like they've broken up with their girlfriend, they don't think, I really want a salad,'' says David. "We wanted orange chicken, something fried, and cold beer. We couldn't find it in Shanghai, so we decided to do it ourselves. When we signed the lease we thought, If this bombs, at least we can eat the food we've been missing for six months."

But it didn't bomb. Fung's family owns 15 Chinese restaurants in the US, the first of which his grandfather set up in Brooklyn in the 60s. Fung flew his dad, who is head chef of all 15 restaurants, over to Shanghai to train up the newly hired Chinese kitchen staff.

"In China they like bones, but we had the staff spend hours deboning the chicken," says Fung. "They were saying, 'Why are we doing this?' We also got them to fill wontons with cheese. They were thinking, What is going on? Some of them were eating cream cheese for the first time. They were shaking their heads."

When Fung transferred his family's fantastic recipes (including a rich orange chicken, Kung Pao chicken, General Tso's beef, and tofu chop suey) to China and served them alongside imported US beers, Western expats latched on quickly. But locals needed to be won over, too—a goal that was achieved when the pair started selling themselves as providing "American food" rather than "Chinese food with an American twist."

That's not to say that things haven't got lost in translation sometimes. "The first response from locals is always about portion size," says Dave. "They think they're huge. We had two petite women come in early on when we opened and order seven dishes. After the second one came out they just started laughing. Also, people hadn't seen the take-out boxes we use anywhere other than on The Big Bang Theory. Our Chinese assistant just said, "'Oh, that's what Sheldon eats.'"

In her talk, Jennifer 8. Lee showed a video of Chinese people looking bemused as they were shown fortune cookies for the first time. (Fortune cookies actually originated from Japan.) The responses have been similar over here: "A lot of our guests are opening their first fortune cookies," says Dave. "Some of them eat the paper or put it in their purse thinking it's a free gift."

Fung believes that it is quality rather than novelty that's earned them the respect of both locals and expats, though. "We're not finding recipes on the internet, we're doing this for real," he says. "Every American-Chinese family has their own recipe for orange chicken, and this is something my grandfather passed on. This food tastes like it does in New York and is legit, with 40 years of history."

As our interview wraps up, Dave hands me a fortune cookie. I break open to reveal a paper slip bearing a message so fitting, I suspect he may have set it up: If you build it, they will come.

Don't expect to get such poignant messages here in the near future. Fung and Dave say they wrote all the fortunes themselves (initially to replace the first batch they ordered that turned out to be written in Dutch), but have run out of ideas. Now they use suggestions written by customers and left in a collection box by the door. "They're always something ridiculously sexual," says Dave. "Or phone numbers with 'For a good time call…' next to them. And, of course, a huge amount of pictures of penises."

Close this section

Nontransitive dice

A set of dice is nontransitive if it contains three dice, A, B, and C, with the property that A rolls higher than B more than half the time, and B rolls higher than C more than half the time, but it is not true that A rolls higher than C more than half the time. In other words, a set of dice is nontransitive if the binary relationX rolls a higher number than Y more than half the time – on its elements is not transitive.

It is possible to find sets of dice with the even stronger property that, for each die in the set, there is another die that rolls a higher number than it more than half the time. Using such a set of dice, one can invent games which are biased in ways that people unused to nontransitive dice might not expect (see Example).


Consider the following set of dice.

  • Die A has sides 2, 2, 4, 4, 9, 9.
  • Die B has sides 1, 1, 6, 6, 8, 8.
  • Die C has sides 3, 3, 5, 5, 7, 7.

The probability that A rolls a higher number than B, the probability that B rolls higher than C, and the probability that C rolls higher than A are all 5/9, so this set of dice is nontransitive. In fact, it has the even stronger property that, for each die in the set, there is another die that rolls a higher number than it more than half the time.

Now, consider the following game, which is played with a set of dice.

  1. The first player chooses a die from the set.
  2. The second player chooses one die from the remaining dice.
  3. Both players roll their die; the player who rolls the higher number wins.

If this game is played with a transitive set of dice, it is either fair or biased in favor of the first player, because the first player can always find a die that will not be beaten by any other dice more than half the time. If it is played with the set of dice described above, however, the game is biased in favor of the second player, because the second player can always find a die that will beat the first player's die with probability 5/9. The following tables show all possible outcomes for all 3 pairs of dice.

Player 1 chooses die A
Player 2 chooses die C
Player 1 chooses die B
Player 2 chooses die A
Player 1 chooses die C
Player 2 chooses die B



2 4 9



1 6 8



3 5 7
3 C A A 2 A B B 1 C C C
5 C C A 4 A B B 6 B B C
7 C C A 9 A A A 8 B B B

Though the three nontransitive dice A, B, C (first set of dice)

  • A: 2, 2, 6, 6, 7, 7
  • B: 1, 1, 5, 5, 9, 9
  • C: 3, 3, 4, 4, 8, 8

P(A > B) = P(B > C) = P(C > A) = 5/9

and the three nontransitive dice A′, B′, C′ (second set of dice)

  • A′: 2, 2, 4, 4, 9, 9
  • B′: 1, 1, 6, 6, 8, 8
  • C′: 3, 3, 5, 5, 7, 7

P(A′ > B′) = P(B′ > C′) = P(C′ > A′) = 5/9

win against each other with equal probability they are not equivalent. While the first set of dice (A, B, C) has a 'highest' die, the second set of dice has a 'lowest' die. Rolling the three dice of a set and using always the highest score for evaluation will show a different winning pattern for the two sets of dice. With the first set of dice, die B will win with the highest probability (88/216) and dice A and C will each win with a probability of 64/216. With the second set of dice, die C′ will win with the lowest probability (56/216) and dice A′ and B′ will each win with a probability of 80/216.

Variations of nontransitive dice[edit]

Efron's dice[edit]

Efron's dice are a set of four nontransitive dice invented by Bradley Efron.

The four dice A, B, C, D have the following numbers on their six faces:

  • A: 4, 4, 4, 4, 0, 0
  • B: 3, 3, 3, 3, 3, 3
  • C: 6, 6, 2, 2, 2, 2
  • D: 5, 5, 5, 1, 1, 1


Each die is beaten by the previous die in the list, with a probability of 2/3:

P(A>B)=P(B>C)=P(C>D)=P(D>A)=23{\displaystyle P(A>B)=P(B>C)=P(C>D)=P(D>A)={2 \over 3}}

B's value is constant; A beats it on 2/3 rolls because four of its six faces are higher.

Similarly, B beats C with a 2/3 probability because only two of C's faces are higher.

P(C>D) can be calculated by summing conditional probabilities for two events:

  • C rolls 6 (probability 1/3); wins regardless of D (probability 1)
  • C rolls 2 (probability 2/3); wins only if D rolls 1 (probability 1/2)

The total probability of win for C is therefore

(13×1)+(23×12)=23{\displaystyle \left({1 \over 3}\times 1\right)+\left({2 \over 3}\times {1 \over 2}\right)={2 \over 3}}

With a similar calculation, the probability of D winning over A is

(12×1)+(12×13)=23{\displaystyle \left({1 \over 2}\times 1\right)+\left({1 \over 2}\times {1 \over 3}\right)={2 \over 3}}

Best overall die[edit]

The four dice have unequal probabilities of beating a die chosen at random from the remaining three:

As proved above, die A beats B two-thirds of the time but beats D only one-third of the time. The probability of die A beating C is 4/9 (A must roll 4 and C must roll 2). So the likelihood of A beating any other randomly selected die is:

13×(23+13+49)=1327{\displaystyle {1 \over 3}\times \left({2 \over 3}+{1 \over 3}+{4 \over 9}\right)={13 \over 27}}

Similarly, die B beats C two-thirds of the time but beats A only one-third of the time. The probability of die B beating D is 1/2 (only when D rolls 1). So the likelihood of B beating any other randomly selected die is:

13×(23+13+12)=12{\displaystyle {1 \over 3}\times \left({2 \over 3}+{1 \over 3}+{1 \over 2}\right)={1 \over 2}}

Die C beats D two-thirds of the time but beats B only one-third of the time. The probability of die C beating A is 5/9. So the likelihood of C beating any other randomly selected die is:

13×(23+13+59)=1427{\displaystyle {1 \over 3}\times \left({2 \over 3}+{1 \over 3}+{5 \over 9}\right)={14 \over 27}}

Finally, die D beats A two-thirds of the time but beats C only one-third of the time. The probability of die D beating B is 1/2 (only when D rolls 5). So the likelihood of D beating any other randomly selected die is:

13×(23+13+12)=12{\displaystyle {1 \over 3}\times \left({2 \over 3}+{1 \over 3}+{1 \over 2}\right)={1 \over 2}}

Therefore, the best overall die is C with a probability of winning of 0.5185. C also rolls the highest average number in absolute terms, 3 1/3. (A's average is 2 2/3, while B's and D's are both 3.)

Variants with equal averages[edit]

Note that Efron's dice have different average rolls: the average roll of A is 8/3, while B and D each average 9/3, and C averages 10/3. The nontransitive property depends on which faces are larger or smaller, but does not depend on the absolute magnitude of the faces. Hence one can find variants of Efron's dice where the odds of winning are unchanged, but all the dice have the same average roll. For example,

  • A: 7, 7, 7, 7, 1, 1
  • B: 5, 5, 5, 5, 5, 5
  • C: 9, 9, 3, 3, 3, 3
  • D: 8, 8, 8, 2, 2, 2

These variant dice are useful, e.g., to introduce students to different ways of comparing random variables (and how only comparing averages may overlook essential details).

Numbered 1 through 24 dice[edit]

A set of four dice using all of the numbers 1 through 24 can be made to be nontransitive. With adjacent pairs, one die will win approximately 2 out of 3 times.

For rolling high number, B beats A, C beats B, D beats C, A beats D.

  • A: 01, 02, 16, 17, 18, 19
  • B: 03, 04, 05, 20, 21, 22
  • C: 06, 07, 08, 09, 23, 24
  • D: 10, 11, 12, 13, 14, 15

Relation to Efron's dice[edit]

These dice are basically the same as Efron's dice, as each number of a series of successive numbers on a single die can all be replaced by the lowest number of the series and afterwards renumbering them.

  • A: 01, 02, 16, 17, 18, 1901, 01, 16, 16, 16, 160, 0, 4, 4, 4, 4
  • B: 03, 04, 05, 20, 21, 2203, 03, 03, 20, 20, 201, 1, 1, 5, 5, 5
  • C: 06, 07, 08, 09, 23, 2406, 06, 06, 06, 23, 232, 2, 2, 2, 6, 6
  • D: 10, 11, 12, 13, 14, 1510, 10, 10, 10, 10, 103, 3, 3, 3, 3, 3

Miwin's dice[edit]

Miwin's Dice were invented in 1975 by the physicist Michael Winkelmann.

Consider a set of three dice, III, IV and V such that

  • die III has sides 1, 2, 5, 6, 7, 9
  • die IV has sides 1, 3, 4, 5, 8, 9
  • die V has sides 2, 3, 4, 6, 7, 8


  • the probability that III rolls a higher number than IV is 17/36
  • the probability that IV rolls a higher number than V is 17/36
  • the probability that V rolls a higher number than III is 17/36

Three-dice set with minimal alterations to standard dice[edit]

The following nontransitive dice have only a few differences compared to 1 through 6 standard dice:

  • as with standard dice, the total number of pips is always 21
  • as with standard dice, the sides only carry pip numbers between 1 and 6
  • faces with the same number of pips occur a maximum of twice per dice
  • only two sides on each die have numbers different from standard dice:
    • A: 1, 1, 3, 5, 5, 6
    • B: 2, 3, 3, 4, 4, 5
    • C: 1, 2, 2, 4, 6, 6

Like Miwin’s set, the probability of A winning versus B (or B vs. C, C vs. A) is 17/36. The probability of a draw, however, is 4/36, so that only 15 out of 36 rolls lose. So the overall winning expectation is higher.

Warren Buffett[edit]

Warren Buffett is known to be a fan of nontransitive dice. In the book Fortune's Formula: The Untold Story of the Scientific Betting System that Beat the Casinos and Wall Street, a discussion between him and Edward Thorp is described. Buffett and Thorp discussed their shared interest in nontransitive dice. "These are a mathematical curiosity, a type of 'trick' dice that confound most people's ideas about probability."

Buffett once attempted to win a game of dice with Bill Gates using nontransitive dice. "Buffett suggested that each of them choose one of the dice, then discard the other two. They would bet on who would roll the highest number most often. Buffett offered to let Gates pick his die first. This suggestion instantly aroused Gates's curiosity. He asked to examine the dice, after which he demanded that Buffett choose first."[1]

In 2010, Wall Street Journal magazine quoted Sharon Osberg, Buffett's bridge partner, saying that when she first visited his office 20 years earlier, he tricked her into playing a game with nontransitive dice that could not be won and "thought it was hilarious".[2]

Nontransitive dice set for more than two players[edit]

A number of people have introduced variations of nontransitive dice where one can compete against more than one opponent.

Three players[edit]

Oskar dice[edit]

Oskar van Deventer introduced a set of seven dice (all faces with probability 1/6) as follows:[3]

  • A: 2, 02, 14, 14, 17, 17
  • B: 7, 07, 10, 10, 16, 16
  • C: 5, 05, 13, 13, 15, 15
  • D: 3, 03, 09, 09, 21, 21
  • E: 1, 01, 12, 12, 20, 20
  • F: 6, 06, 08, 08, 19, 19
  • G: 4, 04, 11, 11, 18, 18

One can verify that A beats {B,C,E}; B beats {C,D,F}; C beats {D,E,G}; D beats {A,E,F}; E beats {B,F,G}; F beats {A,C,G}; G beats {A,B,D}. Consequently, for arbitrarily chosen two dice there is a third one that beats both of them. Namely,

  • G beats {A,B}; F beats {A,C}; G beats {A,D}; D beats {A,E}; D beats {A,F}; F beats {A,G};
  • A beats {B,C}; G beats {B,D}; A beats {B,E}; E beats {B,F}; E beats {B,G};
  • B beats {C,D}; A beats {C,E}; B beats {C,F}; F beats {C,G};
  • C beats {D,E}; B beats {D,F}; C beats {D,G};
  • D beats {E,F}; C beats {E,G};
  • E beats {F,G}.

Whatever the two opponents choose, the third player will find one of the remaining dice that beats both opponents' dice.

Grime dice[edit]

Dr James Grime discovered a set of five dice as follows:[4]

  • A: 2, 2, 2, 7, 7, 7
  • B: 1, 1, 6, 6, 6, 6
  • C: 0, 5, 5, 5, 5, 5
  • D: 4, 4, 4, 4, 4, 9
  • E: 3, 3, 3, 3, 8, 8

One can verify that, when the game is played with one set of Grime dice:

  • A beats B beats C beats D beats E beats A (first chain);
  • A beats C beats E beats B beats D beats A (second chain).

However, when the game is played with two such sets, then the first chain remains the same but the second chain is reversed (i.e. A beats D beats B beats E beats C beats A). Consequently, whatever dice the two opponents choose, the third player can always find one of the remaining dice that beats them both (as long as the player is then allowed to choose between the one-die option and the two-die option):

Dice chosen by the opponents Die one should choose Option one should choose
(1- or 2-die)
A B E 1
A C E 2
A D C 2
A E D 1
B C A 1
B D A 2
B E D 2
C D B 1
C E B 2
D E C 1

There are two major issues with this set, however. The first one is that in the two-die option of the game, the first chain should stay exactly the same in order to make the game nontransitive. In practice, though, D actually beats C. The second problem is that the third player would have to be allowed to choose between the one-die option and the two-die option – which may be seen as unfair to other players.

Corrected Grime dice[edit]

The above issue of D defeating C arises because the dice have 6 faces rather than 5. By replacing the lowest (or highest) face of each die with "reroll" (R), all five dice will function exactly as Dr James Grime intended:

  • A: R, 2, 2, 7, 7, 7
  • B: R, 1, 6, 6, 6, 6
  • C: R, 5, 5, 5, 5, 5
  • D: R, 4, 4, 4, 4, 9
  • E: R, 3, 3, 3, 8, 8

Alternatively, these faces could be mapped to a set of pentagonal-trapezohedral (10-sided) dice, with each number appearing exactly twice, or to a set of icosahedral (20-sided) dice, with each number appearing four times. This eliminates the need for a "reroll" face.

This solution was discovered by Jon Chambers, an Australian Pre-Service Mathematics Teacher.[citation needed]

Four players[edit]

A four-player set has not yet been discovered, but it was proved that such a set would require at least 19 dice.[4]

Nontransitive 4-sided dice[edit]

Tetrahedra can be used as dice with four possible results.

Set 1
  • A: 1, 4, 7, 7
  • B: 2, 6, 6, 6
  • C: 3, 5, 5 ,8

P(A > B) = P(B > C) = P(C > A) = 9/16

The following tables show all possible outcomes:



2 6 6 6
1 B B B B
4 A B B B
7 A A A A
7 A A A A

In "A versus B", A wins in 9 out of 16 cases.



3 5 5 8
2 C C C C
6 B B B C
6 B B B C
6 B B B C

In "B versus C", B wins in 9 out of 16 cases.



1 4 7 7
3 C A A A
5 C C A A
5 C C A A
8 C C C C

In "C versus A", C wins in 9 out of 16 cases.

Set 2
  • A: 3, 3, 3, 6
  • B: 2, 2, 5, 5
  • C: 1, 4, 4, 4

P(A > B) = P(B > C) = 10/16, P(C > A) = 9/16

Nontransitive 12-sided dice[edit]

In analogy to the nontransitive six-sided dice, there are also dodecahedra which serve as nontransitive twelve-sided dice. The points on each of the dice result in the sum of 114. There are no repetitive numbers on each of the dodecahedra.

Miwin’s dodecahedra (set 1) win cyclically against each other in a ratio of 35:34.

The miwin’s dodecahedra (set 2) win cyclically against each other in a ratio of 71:67.

Set 1:

D III with blue dots 1 2 5 6 7 9 10 11 14 15 16 18
D IV with red dots 1 3 4 5 8 9 10 12 13 14 17 18
D V with black dots 2 3 4 6 7 8 11 12 13 15 16 17

Set 2:

D VI with yellow dots 1 2 3 4 9 10 11 12 13 14 17 18
D VII with white dots 1 2 5 6 7 8 9 10 15 16 17 18
D VIII with green dots 3 4 5 6 7 8 11 12 13 14 15 16

Nontransitive prime-numbered 12-sided dice[edit]

It is also possible to construct sets of nontransitive dodecahedra such that there are no repeated numbers and all numbers are primes. Miwin’s nontransitive prime-numbered dodecahedra win cyclically against each other in a ratio of 35:34.

Set 1: The numbers add up to 564.

PD 11 with blue numbers 13 17 29 31 37 43 47 53 67 71 73 83
PD 12 with red numbers 13 19 23 29 41 43 47 59 61 67 79 83
PD 13 with black numbers 17 19 23 31 37 41 53 59 61 71 73 79

Set 2: The numbers add up to 468.

PD 1 with yellow numbers 7 11 19 23 29 37 43 47 53 61 67 71
PD 2 with white numbers 7 13 17 19 31 37 41 43 59 61 67 73
PD 3 with green numbers 11 13 17 23 29 31 41 47 53 59 71 73

See also[edit]



  • Gardner, Martin (2001). The Colossal Book of Mathematics: Classic Puzzles, Paradoxes, and Problems: Number Theory, Algebra, Geometry, Probability, Topology, Game Theory, Infinity, and Other Topics of Recreational Mathematics (1st ed.). New York: W. W. Norton & Company. p. 286–311. [ISBN missing]
  • Spielerische Mathematik mit Miwin'schen Würfeln (in German). Bildungsverlag Lemberger. ISBN 978-3-85221-531-0. 

External links[edit]

Close this section

Wacl: Tcl distro customized for WebAssembly

Tcl distribution for Webassembly or Javascript

This is a Tcl distribution for WebAssembly ( It enables Web developers to embed a Tcl interpreter in the browser and integrate Tcl with JavaScript. It enables Tcl developers to use their tools and language of choice to create client side web applications. It enables all developers to reuse a great and (over decades) grown code base of useful packages and scripts, such as Tcllib, to be used in web browsers.

It is an extension of the Emtcl project from Aidan Hobsen, which can be found here. But Wacl takes things a few steps further: it integrates a fully featured Tcl interpreter into the webpage and adds the following features:

  • A main tclsh interpreter and capability to get it via JavaScript 
  • An event loop to process all Tcl events (timer events, fileevents, custom events)
  • Client sockets. The socket -async ... command connects to websocket servers with the binary protocol. Then the resulting handle can be used to transmit binary data as with normal TCP sockets.
  • The Tcl library: modules and packages in the Emscripten virtual filesystem. You can add your own packages!
  • Proper initialization via Tcl_Init()
  • An extension to call javascript functions from Tcl
  • various useful extensions (see below for a list and comments)
The original illustrative dom command has been moved to the "wasmtcl" namespace in the package of same name, which is available right at startup. This package contains also a command ::wasmtcl::jscall to call javascript functions from Tcl which have been registered before via the jswrap() module function.

The code compiles fine with Emscripten 1.37.9 to JavaScript and WebAssembly. The latter is the preferred format: WebAssembly is only half the size of the JavaScript "asm.js" output (~1.4MB vs. 2.9MB) and at least twice as fast! However, that could induce incompatibilities with older browsers, which don't (yet) support WebAssembly.


The following extensions are included in Wacl

  • wacl native extension with commands wacl::dom and wacl::jscall
  • tDOM for parsing and creating XML and HTML content
  • json and json::write from tcllib
  • html from tcllib
  • javascript from tcllib
  • ncgi (as a dependency for html)
  • rl_json 0.9.7 (Tcl_ObjType for efficiently parsing and creating JSON to/from Tcl dicts)

More extensions can easily be included and used. C extensions can be compiled with Emscripten (with USE_TCL_STUBS disabled and statically initialized via waclAppInit()) and Tcl extensions can be included in the library virtual filesystem.

But be aware that including extensions is a tradeoff: for the additional functionality you pay with a larger download size. The really useful tDOM extension for instance increases the Wacl distribution by not less than 400kB, which must be downloaded  to the users client when (s)he wants to run a wacl based application, and this can be painful with lower bandwidth. Thus it is better to limit the number of packages to what is necessary rather than to build a batteries included distribution which contains everything.

Getting excited

You can try it out here. You can download the precompiled version with the index page to play on your own webpage here. Both of these pages require a recent browser with webassembly support:

  • Mozilla Firefox >= 52.0
  • Google Chrome >= 57.0
  • Microsoft Edge (Windows 10 "Creators" update)
  • Opera

A websocket echo server demo is (tbd).

Getting started

wasmTcl will compile on a Unix/Linux environment with the following tools installed:

  • the Emscripten SDK (installation is documented on the web page)
  • make, autoconf
  • diff, patch
  • fossil (best grab a prebuilt binary, if it doesn't come with your distribution already)

Windows is not supported, but macOS with the appropriate tools from MacPorts will probably work (not tested by myself). First step is to checkout this repository:

fossil clone wacl.fossil
fossil open wacl.fossil

This will checkout the files in the current directory. There is a Makefile with the build steps and a README with instructione. The make procedure does merely download the tcl core sources, apply a small patch and configure & build the interpreter to a webassembly plus accompanying .data + .js files. These files can be deployed to the corresponding web source directories. The Emscripten SDK must be on the PATH (i.e. via source $EMSCRIPTEN/ Once wasmTcl is built, it can be used in any browser which supports webassembly, also on Windows.

The build system can be changed to produce javascript instead of webassembly, by simply removing the -s WASM=1 flag from the BCFLAGS variable in the Makefile. This will generate a larger (~2.8MB), yet minified .js output, which is slower at runtime, but compatible with browsers that don't support webassembly.

Close this section

Rust your ARM microcontroller

Want to program your microcontroller in Rust but your microcontroller vendor doesn’t provide a Rust HAL / SDK? No wonder. AFAIK, no vendor is betting for Rust … yet. How about binding to a C HAL? No? Don’t feel like wrestling with bindgen and the HAL build system and then having a bunch of unsafe FFI calls in your application? OK, how’s this alternative: A method to easily build 100% Rust applications that can use all the device hardware through a memory safe API? Sounds good? Excellent because that’s today menu.

In this post, I’ll cover how to build an application for a Cortex-M microcontroller from scratch. However, I’m going to take a practical approach here and omit explaining low level details like linker scripts and the boot sequence. There are crates published on that deal with those low level details so we’ll leverage those instead of reinventing the wheel.


For this demo, I’m going to use the STM32F3DISCOVERY development board but the steps here can be adapted to any other development board. Here are the specifications of the DISCOVERY board:

  • Microcontroller: STM32F303VCT6
  • Core: ARM Cortex-M4 + FPU
  • RAM: 40+8 KiB
  • Flash: 256 KiB
  • Peripherals: Timers, Serial, I2C, SPI, PWM, etc.

We’ll need these tools on the host system:

  • A nightly Rust toolchain.
  • Xargo, to build the core crate on the fly.
  • A linker: GNU ld .
  • A debugger: GDB .
  • OpenOCD, to communicate with the in-circuit / external programmer. (The DISCOVERY board has a built-in SWD based programmer)

Installation instructions for Arch Linux:

$ # Switch to the nightly channel
$ rustup default nightly

$ rustc -V
rustc 1.18.0-nightly (2bd4b5c6d 2017-04-23)

$ sudo pacman -S arm-none-eabi-binutils arm-none-eabi-gdb openocd

$ arm-none-eabi-ld -V | head -n1
GNU ld (GNU Binutils) 2.28

$ arm-none-eabi-gdb -v | head -n1
GNU gdb (GDB) 7.12.1

$ openocd -v 2>&1 | head -n1
Open On-Chip Debugger 0.10.0

$ cargo install xargo

$ xargo -V
xargo 0.3.6
cargo 0.19.0-nightly (8326a3683 2017-04-19)

$ # for Xargo
$ rustup component add rust-src

You can find installation instructions for Windows and macOS here.

We’ll use the cortex-m-quickstart crate as a template ; it contains all the pieces needed to build a microcontroller application. This template should work for any microcontroller that lets you override the boot sequence . I have tested this template with 6 different microcontrollers / development boards from 3 different vendors without a hitch.

$ # if you don't have the `clone` subcommand
$ cargo install cargo-clone

$ cargo clone cortex-m-quickstart --vers 0.1.1

$ mv cortex-m-quickstart demo && cd $_

$ # change project name and author
$ edit Cargo.toml && head $_
authors = ["Jorge Aparicio <>"]
name = "demo"
version = "0.1.0"

Each microcontroller has different amounts of RAM and Flash memory, and the location of these memory regions in the address space can vary from vendor to vendor. We have to specify this information in the memory.x file to produce a binary that’s valid for the target device. For this demo, I’ll use this file :

$ edit memory.x && cat $_
  /* NOTE K = KiBi = 1024 bytes */
  FLASH : ORIGIN = 0x08000000, LENGTH = 256K
  RAM : ORIGIN = 0x20000000, LENGTH = 40K

/* NOTE Do NOT modify `_stack_start` unless you know what you are doing */
_stack_start = ORIGIN(RAM) + LENGTH(RAM);

The memory layout of the program will look like this:

Memory layout

The .bss + .data region is where static variables are stored. The size of this region is known at compile time and doesn’t change at runtime. The call stack region can grow or shrink at runtime due to function calls. There’s no heap .

Finally, for convenience we set a default target in .cargo/config. With this we can omit the --target flag on every Xargo invocation.

$ cat >>.cargo/config <<'EOF'
target = "thumbv7em-none-eabihf"

The target chosen here must match the ARM core inside the target device. There are four options:

  • thumbv6m-none-eabi, for Cortex M0 and M0+ devices.
  • thumbv7m-none-eabi, for Cortex M3 devices.
  • thumbv7em-none-eabi, for Cortex M4 and M7 devices. No FPU.
  • thumbv7em-none-eabihf, for Cortex M4 and M7 devices. With FPU.

We are done setting up the template. Let’s build examples/ as our first program. This program will print "Hello, world!" on the host console. This program is written in a device agnostic manner and will work on any microcontroller. The only requirement to be able to see the message on the host side is a GDB connection between host and device.

For convenience, here’s the full source of the program:

//! Prints "Hello, world!" on the OpenOCD console using semihosting


extern crate cortex_m;
extern crate cortex_m_rt;

use cortex_m::asm;

fn main() {
    hprintln!("Hello, world!");

// As we are not using interrupts, we just register a dummy catch all handler
#[link_section = ".rodata.interrupts"]
static INTERRUPTS: [extern "C" fn(); 240] = [default_handler; 240];

extern "C" fn default_handler() {

At first glance, it looks kind of normal – it has a main function like the programs that link to std. Let’s inspect it in parts.


We don’t link to std as std doesn’t support microcontrollers.

extern crate cortex_m;
extern crate cortex_m_rt;

The cortex-m-rt crate is a very small runtime that boots the device, initializes RAM and then calls main. It does all this implicitly; you only need to link to it with extern crate to opt into this runtime. The cortex-m crate provides an API to use functionality common to all Cortex-M microcontrollers.

fn main() {
    hprintln!("Hello, world!");

In main, we use the hprintln! macro from the cortex-m crate to print the message to the OpenOCD console . The syntax is the same as println! in std.

#[link_section = ".rodata.interrupts"]
static INTERRUPTS: [extern "C" fn(); 240] = [default_handler; 240];

There’s also this INTERRUPTS variable. This variable is used to register interrupt handlers. As we are not using any interrupt we just register a catch all handler for all the possible interrupt sources.

Let’s now build this program.

$ xargo build --example hello

$ arm-none-eabi-readelf -A target/thumbv7em-none-eabihf/debug/examples/hello                          <<<
Attribute Section: aeabi
File Attributes
  Tag_conformance: "2.09"
  Tag_CPU_arch: v7E-M
  Tag_CPU_arch_profile: Microcontroller
  Tag_THUMB_ISA_use: Thumb-2
  Tag_FP_arch: VFPv4-D16
  Tag_ABI_PCS_GOT_use: direct
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_HardFP_use: SP only
  Tag_ABI_VFP_args: VFP registers
  Tag_ABI_optimization_goals: Prefer Debug
  Tag_CPU_unaligned_access: v6
  Tag_FP_HP_extension: Allowed
  Tag_ABI_FP_16bit_format: IEEE 754

And then run it on the microcontroller.

$ # On one terminal. Leave this running
$ openocd -f interface/stlink-v2-1.cfg -f target/stm32f3x.cfg
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v27 API v2 SWIM v15 VID 0x0483 PID 0x374B
Info : using stlink api v2
Info : Target voltage: 2.920499
Info : stm32f3x.cpu: hardware has 6 breakpoints, 4 watchpoints
$ # On another terminal
$ # Enable safe loading of .gdbinit scripts
$ echo 'set auto-load safe-path /' >> ~/.gdbinit

$ arm-none-eabi-gdb target/thumbv7em-none-eabihf/debug/examples/hello

Thanks to the .gdbinit in the root of the Cargo project, the debugger will drop you at the entry point of the program, which is where the program starts its execution. In the source view, you can see that the cortex-m-rt runtime does what I said it does.

GDB session

(gdb-dashboard, if you were wondering “Hey, what’s that cool GDB UI?”)

From that point you can step through the program all the way to main but the fastest way to get there is to set a breakpoint and let the program run:

> # On the GDB shell
> tbreak hello::main
> continue

You should see:

At the `main` function

After you step over the hprintln! line, you should see this on the OpenOCD terminal:

$ # On the OpenOCD terminal
Info : halted: PC: 0x08000a30
Hello, world!
Info : halted: PC: 0x08000414

There you go: An embedded “Hello, world” in Rust.

Some of you may be wondering whats happens when main returns since embedded programs are supposed to never end. What the runtime does is put the device in “reactive” mode (loop { asm!("wfi") }), where it services interrupts and then sleeps when there’s nothing to do.

The cortex-m-rt ships with a few extra optional features that result in a more pleasant development experience. I think they are just too good to miss the opportunity of showing them to you so let me show two use cases where they come in handy:

Debugging an exception

Consider this program: (See examples/ for the full source).

fn main() {
    // Read an invalid memory address
    unsafe {
        ptr::read_volatile(0x2FFF_FFFF as *const u32);

It tries to read an invalid memory address. Although this is kinda obvious from the source code (if you know what typical RAM addresses look like), let’s see how the runtime would have helped us debug this problem.

If you debug this program and just leave it run freely, you’ll end with a debug session like the one shown below.

GDB session

Fatal errors like this invalid memory access are handled by the hardware through an exception mechanism. When such an error occurs, the processor stops doing whatever it was doing and immediately calls the corresponding exception handler, which is just a function from the POV of the programmer.

The cortex-m-rt crate injects a catch all exception handler tailored for debugging. This handler will trigger a breakpoint during an exception and give you access to plenty of information about the cause of the exception.

Continuing the debug session: A backtrace will give us a general idea of where we came from:

> # Where did we come from?
> backtrace
#0  cortex_m::exception::default_handler::handler (_sr=0x20009f50) at $CARGO_HOME/registry/src/
#1  <signal handler called>
#2  core::ptr::read_volatile<u32> (src=0x2fffffff) at $SYSROOT/lib/rustlib/src/rust/src/libcore/
#3  0x08000442 in crash::main () at $PWD/examples/
#4  0x08000684 in cortex_m_rt::lang_items::start (main=0x8000435 <crash::main>, _argc=0, _argv=0x0) at $CARGO_HOME/registry/src/
#5  0x08000484 in main ()

<signal handler called> is the hardware calling the exception handler, so read_volatile is where the exception occurred.

Within the exception handler context, there’s this Exception value which indicates what kind of exception was raised.

> # What exception did we hit?
> p _e
$1 = cortex_m::exception::Exception::HardFault

There are different kinds of exceptions and each one has its own handler. The hard fault exception is the exception that’s called when a invalid memory access occurs or when the processor tries to execute an invalid instruction, among other fatal errors.

There’s also this StackedRegisters value, this is a snapshot of the CPU registers at the time the exception occurred.

> # What was the state of the program when the exception occurred?
> print/x *_sr
$2 = cortex_m::exception::StackedRegisters {
  r0: 0x2fffffff,
  r1: 0x2fffffff,
  r2: 0x0,
  r3: 0x0,
  r12: 0x0,
  lr: 0x8000427,
  pc: 0x8000408,
  xpsr: 0x61000200

Perhaps the most important of these registers is the pc (Program Counter) register; it points to the instruction that triggered the exception. We can disassemble the program around that instruction to investigate further:

> # What instruction generated the exception?
> disassemble /m _sr.pc
Dump of assembler code for function core::ptr::read_volatile<u32>:
330     pub unsafe fn read_volatile<T>(src: *const T) -> T {
   0x08000400 <+0>:     sub     sp, #20
   0x08000402 <+2>:     mov     r1, r0
   0x08000404 <+4>:     str     r0, [sp, #8]
   0x08000406 <+6>:     str     r0, [sp, #12]

331         intrinsics::volatile_load(src)
   0x08000408 <+8>:     ldr     r0, [r0, #0]
   0x0800040a <+10>:    str     r0, [sp, #16]
   0x0800040c <+12>:    str     r1, [sp, #4]
   0x0800040e <+14>:    str     r0, [sp, #0]
   0x08000410 <+16>:    b.n     0x8000412 <core::ptr::read_volatile<u32>+18>

332     }
   0x08000412 <+18>:    ldr     r0, [sp, #0]
   0x08000414 <+20>:    add     sp, #20
   0x08000416 <+22>:    bx      lr

End of assembler dump.

0x08000408: ldr r0, [r0, #0] is pointed out as the culprit. This instruction tries to load the value stored at the address indicated by the r0 register. From the StackedRegisters value, we know that r0 was 0x2fffffff. This must mean that the address 0x2fffffff is invalid as in it must not be in the RAM or Flash memory region. Which is exactly the case here.

Redirection of panic! messages

Another nifty thing that the cortex-m-rt crate can do is print panic! messages on the OpenOCD console just like the hprintln! macro does.

Consider this program (See examples/ for the full source).

fn main() {

If we enable the panic-over-semihosting feature of the cortex-m-rt crate, we’ll see the panic! message appear on the OpenOCD console when the program is executed under the debugger.

$ # On the OpenOCD console
Info : halted: PC: 0x0800160c
panicked at 'Oops', examples/

Line and source file information will show up just like they do when a regular Rust program panics. This is pretty useful to catch bugs like arithmetic overflow and out of bounds accesses. Oh, and panic!s trigger breakpoints just like exceptions so you can use backtrace within the GDB session to get a backtrace.

Now that we have verified that both the tooling and the template work, we can go ahead and build a program that makes use of the device hardware.

As you may know, using the hardware requires reading and writing to special memory regions referred to as registers. The reference manual of the microcontroller contains all there is to know about a microcontroller’s registers: their locations in memory and what their contents mean. You could translate that information into an API but that’s a lot of work and error prone.

A better way is to auto-generate that API from the microcontroller’s System View Description (SVD) file using a tool like svd2rust. A SVD file is basically a machine readable version of the reference manual. Most vendors provide these for their devices. Here’s a database of such files – it contains more than 500 SVD files. If you don’t find a SVD file for your device there, check your microcontroller vendor website or ask them directly.

Let’s use svd2rust on the SVD file of the STM32F303VCT6 microcontroller .

$ cd ..

$ cargo new --lib stm32f30x && cd $_

$ # Fetch the SVD from the database
$ curl -LO
$ dos2unix STM32F30x.svd

$ # Patch the SVD for extra type safety
$ curl -L | patch -p1

$ cargo install svd2rust --vers 0.7.0

$ # Turn the SVD file into a device crate
$ svd2rust -i STM32F30x.svd | rustfmt > src/

$ # Wow, you certainly don't want to write all that by hand!
$ wc src/
 226424 1153424 7689220 src/

$ # if you don't have the `add` subcommand
$ cargo install cargo-edit

$ # Dependencies of the device crate
$ cargo add cortex-m vcell

$ # sanity check
$ xargo build --target thumbv7em-none-eabihf

That wasn’t too hard and 200K+ lines of Rust just materialized from thin air.

The output of svd2rust is a crate that provides an API to access every one of the microcontroller’s peripherals. The API is relatively low level as it operates at the register level but it’s type safe: It won’t let you

  • Write to read-only registers.
  • Read write-only registers.
  • Read or write to the reserved parts of a register.
  • Write invalid bit patterns to a register. For example, the 2 lowest bits of a register may only support the values 0b01, 0b10 and 0b11 – the API enforces that those are the only values that you can write to those two bits.

Apart from avoiding those footguns, the API uses enums / methods instead of magic bit patterns like 0b01 for clarity. The generated API is documented here.

Armed with an API to access the hardware, we now can write the real “Hello, world!” of the embedded world: A program to blink an LED.

There’s no example for this in cortex-m-quickstart as the implementation is device and board specific but here’s the code for the STM32F3DISCOVERY:

// examples/


// version = "0.2.0", default-features = false
extern crate cast;
extern crate cortex_m;
extern crate cortex_m_rt;
extern crate stm32f30x;

use core::u16;

use cast::{u16, u32};
use cortex_m::asm;
use stm32f30x::{GPIOE, RCC, TIM7};

mod frequency {
    /// Frequency of APB1 bus (TIM7 is connected to this bus)
    pub const APB1: u32 = 8_000_000;

/// Timer frequency
const FREQUENCY: u32 = 1;

fn main() {
    // Critical section, this closure is non-preemptable
        |cs| {
            // Exclusive access to the peripherals
            let gpioe = GPIOE.borrow(cs);
            let rcc = RCC.borrow(cs);
            let tim7 = TIM7.borrow(cs);

            // Power up the relevant peripherals
            rcc.ahbenr.modify(|_, w| w.iopeen().enabled());
            rcc.apb1enr.modify(|_, w| w.tim7en().enabled());

            // Configure the pin PE9 as an output pin
            gpioe.moder.modify(|_, w| w.moder9().output());

            // Configure TIM7 for periodic timeouts
            let ratio = frequency::APB1 / FREQUENCY;
            let psc = u16((ratio - 1) / u32(u16::MAX)).unwrap();
            tim7.psc.write(|w| w.psc().bits(psc));
            let arr = u16(ratio / u32(psc + 1)).unwrap();
            tim7.arr.write(|w| w.arr().bits(arr));
            tim7.cr1.write(|w| w.opm().continuous());

            // Start the timer
            tim7.cr1.modify(|_, w| w.cen().enabled());

            // APPLICATION LOGIC
            let mut state = false;
            loop {
                // Wait for an update event
                while {}

                // Clear the update event flag
      |_, w| w.uif().clear());

                // Toggle the state
                state = !state;

                // Blink the LED
                if state {
                    gpioe.bsrr.write(|w| w.bs9().set());
                } else {
                    gpioe.bsrr.write(|w| w.br9().reset());

// This part is the same as before
#[link_section = ".rodata.interrupts"]
static INTERRUPTS: [extern "C" fn(); 240] = [default_handler; 240];

extern "C" fn default_handler() {

It’s not necessary to understand every single line of the previous program as it contains low level device specific code. But here are some things to note:

  • There’s no unsafe code! Peripherals are global resources and microcontrollers have hardware support for preemption in the form of interrupts and exceptions so unsynchronized access to a peripheral is unsafe in the general case. Here we add synchronization in the form of a critical section (interrupt::free) which ensures that the whole closure is executed “atomically”, i.e. without being interrupted.

  • Once we are inside a critical section, we can safely borrow / access the peripherals GPIOE, RCC and TIM7 for the duration of the critical section.

  • The program never ends; there’s an infinite loop that prevents that.

  • The timer is configured to generate an update event every second. The timer notifies the processor about the update event by setting an update event flag, which is just a bit at some known memory location. In this program, we continuously check for the state of the flag to force the processor to wait for 1 second before toggling the state of the LED. This continuous polling approach is known as busy waiting.

And here’s the outcome:

$ # depend on the previously generated device crate
$ cargo add stm32f30x --path ../stm32f30x

$ xargo build --example blinky

$ arm-none-eabi-gdb target/thumbv7em-none-eabihf/debug/examples/blinky

Yay, it works!

But the best part is the disassembly of the program when compiled in release mode.

$ xargo build --example blinky --release

$ arm-none-eabi-objdump -Cd target/thumbv7em-none-eabihf/release/examples/blinky
08000400 <blinky::main>:
 8000400:	b580      	push	{r7, lr}
 8000402:	f3ef 8010 	mrs	r0, PRIMASK
 8000406:	b672      	cpsid	i
 8000408:	2201      	movs	r2, #1
 800040a:	2300      	movs	r3, #0
 800040c:	f04f 7c00 	mov.w	ip, #33554432	; 0x2000000
 8000410:	f44f 7e00 	mov.w	lr, #512	; 0x200
 8000414:	f241 0014 	movw	r0, #4116	; 0x1014
 8000418:	f2c4 0002 	movt	r0, #16386	; 0x4002
 800041c:	6801      	ldr	r1, [r0, #0]
 800041e:	f441 1100 	orr.w	r1, r1, #2097152	; 0x200000
 8000422:	6001      	str	r1, [r0, #0]
 8000424:	6881      	ldr	r1, [r0, #8]
 8000426:	f041 0120 	orr.w	r1, r1, #32
 800042a:	6081      	str	r1, [r0, #8]
 800042c:	f241 0000 	movw	r0, #4096	; 0x1000
 8000430:	f6c4 0000 	movt	r0, #18432	; 0x4800
 8000434:	6801      	ldr	r1, [r0, #0]
 8000436:	f362 4193 	bfi	r1, r2, #18, #2
 800043a:	227a      	movs	r2, #122	; 0x7a
 800043c:	6001      	str	r1, [r0, #0]
 800043e:	f241 4100 	movw	r1, #5120	; 0x1400
 8000442:	f2c4 0100 	movt	r1, #16384	; 0x4000
 8000446:	628a      	str	r2, [r1, #40]	; 0x28
 8000448:	f64f 6210 	movw	r2, #65040	; 0xfe10
 800044c:	62ca      	str	r2, [r1, #44]	; 0x2c
 800044e:	600b      	str	r3, [r1, #0]
 8000450:	680a      	ldr	r2, [r1, #0]
 8000452:	f042 0201 	orr.w	r2, r2, #1
 8000456:	600a      	str	r2, [r1, #0]
 8000458:	e00d      	b.n	8000476 <blinky::main+0x76>
 800045a:	690a      	ldr	r2, [r1, #16]
 800045c:	f013 0f01 	tst.w	r3, #1
 8000460:	f022 0201 	bic.w	r2, r2, #1
 8000464:	610a      	str	r2, [r1, #16]
 8000466:	f083 0201 	eor.w	r2, r3, #1
 800046a:	bf14      	ite	ne
 800046c:	f8c0 c018 	strne.w	ip, [r0, #24]
 8000470:	f8c0 e018 	streq.w	lr, [r0, #24]
 8000474:	4613      	mov	r3, r2
 8000476:	690a      	ldr	r2, [r1, #16]
 8000478:	f012 0f01 	tst.w	r2, #1
 800047c:	d0fb      	beq.n	8000476 <blinky::main+0x76>
 800047e:	e7ec      	b.n	800045a <blinky::main+0x5a>

The svd2rust generated API makes heavy use of closures and enums for type safety, and of method chains for ergonomics but LLVM optimizes all that away and produces very lean code. Also notice how there are no panic! branches; LLVM proved that the failure branches of the unwrap methods were unreachable and optimized them away. This shows that the type safety provided by the svd2rust generated API is zero cost.

Although you could go ahead and build applications just using the register level API that svd2rust generates, it’s nicer if you can code in something higher level and that abstracts away the registers. That’s where Board Support Crates come in.

A board support crate provides a high level API to program a specific development board. I have developed one such crate for the STM32F3DISCOVERY: the f3 crate. Let’s use that crate to simplify the blinky program.

// examples/


// version = "0.2.4"
extern crate cortex_m;

// version = "0.2.0"
extern crate cortex_m_rt;

// version = "0.4.0"
extern crate f3;

use cortex_m::asm;
use f3::led::{self, LEDS};
use f3::stm32f30x::{GPIOE, RCC, TIM7};
use f3::timer::Timer;

/// Timer frequency
const FREQUENCY: u32 = 1;

fn main() {
    // Critical section
        |cs| {
            // Exclusive access to the peripherals
            let gpioe = GPIOE.borrow(cs);
            let rcc = RCC.borrow(cs);
            let tim7 = TIM7.borrow(cs);

            // Configure the PEx pins as output pins
            led::init(gpioe, rcc);

            // Configure TIM7 for periodic timeouts
            let timer = Timer(tim7);
            timer.init(rcc, FREQUENCY);

            // Start the timer

            let mut state = false;
            loop {
                // Wait for an update event *and* clear the update event flag
                while timer.clear_update_flag().is_err() {}

                // Toggle the state
                state = !state;

                // Blink the LED
                if state {
                } else {


// This part is the same as before
#[link_section = ".rodata.interrupts"]
static INTERRUPTS: [extern "C" fn(); 240] = [default_handler; 240];

extern "C" fn default_handler() {

All the register names are gone. The Timer abstraction encodes the idea of a timer that generates periodic update events. Even the LEDs have their own abstraction in the form of the Led type. And LEDS is a collection of all the user LEDs on the board.

Behavior is unchanged after moving to the f3 crate so I’m not going to show another video of the same thing. Let’s see how the new program does in terms of performance.

$ # Remove the device crate
$ cargo rm stm32f30x

$ # Add the board support crate
$ cargo add f3 --vers 0.4.0

$ # the f3 crate provides a memory.x file so we must the remove the one we have
$ rm memory.x

$ xargo build --example blinky2 --release

$ arm-none-eabi-objdump -Cd target/thumbv7em-none-eabihf/release/examples/blinky
08000400 <blinky2::main>:
 8000400:	b580      	push	{r7, lr}
 8000402:	f241 0114 	movw	r1, #4116	; 0x1014
 8000406:	f3ef 8010 	mrs	r0, PRIMASK
 800040a:	b672      	cpsid	i
 800040c:	2300      	movs	r3, #0
 800040e:	f04f 7c00 	mov.w	ip, #33554432	; 0x2000000
 8000412:	f44f 7e00 	mov.w	lr, #512	; 0x200
 8000416:	f2c4 0102 	movt	r1, #16386	; 0x4002
 800041a:	6808      	ldr	r0, [r1, #0]
 800041c:	f440 1000 	orr.w	r0, r0, #2097152	; 0x200000
 8000420:	6008      	str	r0, [r1, #0]
 8000422:	f241 0000 	movw	r0, #4096	; 0x1000
 8000426:	f6c4 0000 	movt	r0, #18432	; 0x4800
 800042a:	6802      	ldr	r2, [r0, #0]
 800042c:	f2c5 5255 	movt	r2, #21845	; 0x5555
 8000430:	6002      	str	r2, [r0, #0]
 8000432:	688a      	ldr	r2, [r1, #8]
 8000434:	f042 0220 	orr.w	r2, r2, #32
 8000438:	608a      	str	r2, [r1, #8]
 800043a:	f241 4100 	movw	r1, #5120	; 0x1400
 800043e:	227a      	movs	r2, #122	; 0x7a
 8000440:	f2c4 0100 	movt	r1, #16384	; 0x4000
 8000444:	628a      	str	r2, [r1, #40]	; 0x28
 8000446:	f64f 6210 	movw	r2, #65040	; 0xfe10
 800044a:	62ca      	str	r2, [r1, #44]	; 0x2c
 800044c:	2201      	movs	r2, #1
 800044e:	60ca      	str	r2, [r1, #12]
 8000450:	600b      	str	r3, [r1, #0]
 8000452:	680a      	ldr	r2, [r1, #0]
 8000454:	f042 0201 	orr.w	r2, r2, #1
 8000458:	600a      	str	r2, [r1, #0]
 800045a:	e00d      	b.n	8000478 <blinky2::main+0x78>
 800045c:	690a      	ldr	r2, [r1, #16]
 800045e:	f013 0f01 	tst.w	r3, #1
 8000462:	f022 0201 	bic.w	r2, r2, #1
 8000466:	610a      	str	r2, [r1, #16]
 8000468:	f083 0201 	eor.w	r2, r3, #1
 800046c:	bf14      	ite	ne
 800046e:	f8c0 c018 	strne.w	ip, [r0, #24]
 8000472:	f8c0 e018 	streq.w	lr, [r0, #24]
 8000476:	4613      	mov	r3, r2
 8000478:	690a      	ldr	r2, [r1, #16]
 800047a:	f012 0f01 	tst.w	r2, #1
 800047e:	d0fb      	beq.n	8000478 <blinky2::main+0x78>
 8000480:	e7ec      	b.n	800045c <blinky2::main+0x5c>

The generated code is about the same as the version that directly used the stm32f30x crate. So the higher level API provided by the f3 crate is also a zero cost abstraction.

OK. That’s all the code I’m going to show you in this post.

I’d like to step back for a bit to look at all the abstraction layers (crates) we are using:

Monotask application

In the above diagram, you can see that the layers are split in two halves. The layers on the left are what I like to call device agnostic layers. The crates on that side rely on hardware features available on all Cortex-M cores so they work for any microcontroller. The crates on the right side are device specific and will only work on a limited set of microcontrollers.

Here’s a recap of what each crate does:

  • cortex-m-rt. It handles the boot process and provides debugging facilities. This crate can’t be directly accessed by the application as there’s no API. This crate is device agnostic.

  • cortex-m. API to use Cortex-M functionality available to all microcontrollers.

  • stm32f30x, the device crate. Hardware access API specific to a device family. Operates at a register level.

  • f3, the Board Support Crate. It provides a higher level API to use the microcontroller peripherals and it’s tailored for a specific development board. This crates builds on top of the device crate.

That’s it for this post. We have seen:

  • A template that handles all the low level stuff: the linker script, linker arguments, sysroot customization and boot sequence. I didn’t even mention the linker arguments and sysroot customization up there because you didn’t need to know about them but the template handled that as well.

  • A tool to generate an API to access all the hardware of a microcontroller.

  • How to write memory safe (zero unsafe) single task applications, using critical sections (interrupt::free).

In the next one, we’ll fix this inefficiency:

                // Wait for an update event and clear the flag
                while timer.clear_update_flag().is_err() {}

This is blocking code that forces the processor to uselessly busy wait for a whole second when it could have been doing something more useful. Also, although critical sections give the synchronization needed to achieve memory safety, they should be used sparingly. Here we executed everything inside a critical section. We’ll fix that too. So in the next post, with just one more device agnostic crate, we’ll leverage interrupts in a memory safe manner and build an efficient multitasking application. I’ll leave you with this figure.

Multitask application

Let’s discuss on reddit.

Enjoyed this post? Like my work on embedded stuff? Consider supporting me on Patreon!

Follow me on twitter for even more embedded stuff.

Close this section

Tiny Linux distro that runs the entire OS as Docker containers

The smallest, easiest way to run Docker in production at scale. Everything in RancherOS is a container managed by Docker. This includes system services such as udev and rsyslog. RancherOS includes only the bare minimum amount of software needed to run Docker. This keeps the binary download of RancherOS very small. Everything else can be pulled in dynamically through Docker.

How this works

Everything in RancherOS is a Docker container. We accomplish this by launching two instances of Docker. One is what we call the system Docker which runs as the first process. System Docker then launches a container that runs the user Docker. The user Docker is then the instance that gets primarily used to create containers. We created this separation because it seemed logical and also it would really be bad if somebody did docker rm -f $(docker ps -qa) and deleted the entire OS.

How it works

Latest Release

v1.0.1 - Docker 17.03.1-ce - Linux 4.9.24


Additional Downloads

Latest Links

v1.0.1 Links

v1.0.0 Links - updates coming soon

Note: you can use http instead of https in the above URLs, e.g. for iPXE.


SSH keys are added to the rancher user, so you must log in using the rancher user.


Google Compute Engine

We are providing a disk image that users can download and import for use in Google Compute Engine. The image can be obtained from the release artifacts for RancherOS.

Download Image

Please follow the directions at our docs to launch in GCE.

Documentation for RancherOS

Please refer to our RancherOS Documentation website to read all about RancherOS. It has detailed information on how RancherOS works, getting-started and other details.

Support, Discussion, and Community

If you need any help with RancherOS or Rancher, please join us at either our Rancher forums or #rancher IRC channel where most of our team hangs out at.

For security issues, please email instead of posting a public issue in GitHub. You may (but are not required to) use the GPG key located on Keybase.

Please submit any RancherOS bugs, issues, and feature requests to rancher/os.

Please submit any Rancher bugs, issues, and feature requests to rancher/rancher.

#License Copyright (c) 2014-2017 Rancher Labs, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Close this section

How Web Forums Make Neuroticism Viral

Web forums are full of neurotics. They breed neuroticism and make it go viral. Learn how and why this happens.

Open Discussion On The Web

One of the great virtues of the internet is that it enables open discussion, globally. There’s a web forum for pretty much any niche you can imagine, from communism to Nazism, from basket-weaving to base jumping.

No matter how niche the interest, the web can aggregate all devotees in one place. This is the genius of Reddit, now the 4th most-visited site in the US and #9 globally. Anybody can create a subreddit for any interest they have. Facebook users can create large global groups (at great cost to society). Twitter users create hashtags and linked networks of users for particular interests.

You can find a place to talk about anything you want on the internet. And if you can’t, make it yourself.


A quick note on “neuroticism” as I refer to it here. I mean it in the sense of the “Big Five” personality traits. I’m going to lift a quick definition straight from Wikipedia, which will serve our purposes:

Individuals who score high on neuroticism are more likely than average to be moody and to experience such feelings as anxiety, worry, fear, anger, frustration, envy, jealousy, guilt, depressed mood, and loneliness. People who are neurotic respond worse to stressors and are more likely to interpret ordinary situations as threatening and minor frustrations as hopelessly difficult.

Neuroticism Is Widespread In Web Forums

A strange thing tends to happen as web communities grow: they become flooded with neurotics.

Let’s take a look at a couple examples to illustrate the point. If you’re a new parent or know someone who is, you may have heard of Mumsnet. The site began as a small discussion forum for mothers in the UK (hence the spelling). But a lack of quality web resources on parenting meant that it quickly showed up, ranked highly, in Google searches for all sorts of common parenting problems. The site blossomed into a behemoth; it’s one of the largest parenting sites in the world and the leader for discussion among parents.

Another thing: it’s packed to the gills with neurotic mothers. Take a look at the Mumsnet Madness Twitter feed for some examples. I picked one at random to share:

Imagine showing some of these posts to an average mother from the 1960s.

Another example, a little more personal. As a teenager graduating high school, I applied to US universities for a shot at the Ivy League. Nobody I knew had ever moved abroad for an education, and nobody had in the history of my high school. Having to do the research myself, I turned to web forums for guidance. I ended up on College Confidential, self-described as “the world’s largest college forums”. This is the place!, I thought to myself. This is where I’ll learn about what I need to do to get in, and how to structure my application.

As with Mumsnet: the place is chock full of absolute nutjobs. Take College Confidential’s word for it, and you’ll need a 5.0 GPA with a perfect SAT, a few patents, eight verifiable and quantified extracurricular activities, and perhaps some published papers just to sniff the Ivy League. Practically every post is high schoolers shitting themselves with anxiety or putting others down. Nobody believes in themselves, but they know that not getting in to Harvard is a death sentence for any hope of success in life.

I ended up ignoring College Confidential and didn’t have any issues. I imagine parenting is easier, too, without online hyperventilation about whether balloons are safe.

Why Do Neurotics Prevail In Online Communities?

Why do these communities end up dominated by weirdos? Especially the niche ones?

The reason is time. It is weirdos and neurotics that are most disposed to spending all their time posting online, so they end up generating the majority of the content.

Think about it. If you’re a successful parent, why would you spend hours a day on Mumsnet? Your kids are fine, and if your mental health is, too, you’ve probably got better things to do than spend your time on a web forum. If you’re a talented artist, you probably spend your time actually making art rather than discussing technique online. Top amateur athletes are too busy actually training and playing to spend five hours a day on a forum for their hobby.

The people actually taking action are walking the walk. The people talking are doing just that. And, given their lack of action, the talkers have more time to talk.

Most of the content is produced by the talkers, not the walkers. Some communities have adopted the term “keyboard warriors” to refer to those who are all theory and no practice. They end up dominating the discussion.

Another reason is that neuroticism is very compatible with the format of web forums. Remember the definition we used above? People with neurotic tendencies don’t fare too well in real-world scenarios, and so they take refuge in online communities. 4chan is famously proud of its “autists”.

You end up with two forces pushing neuroticism in web communities: the greater propensity of neurotics to post, and the suitability of web forums to neurotic personality traits.

Viral Neurosis

Online communities don’t just foster neuroticism; they make it go viral.

It would be fine if web forums provided a space for neurotics to communicate among one another. But remember how Mumsnet grew: the volume of content they had brought them to the top of every Google search for parenting. Remember how I found College Confidential: they were “the world’s biggest college forum”.

Neurotics spur the growth of online forums through pure volume of content. Regular people get dragged in to these communities through web searching.

You’re probably familiar with some version of this effect if you’ve done any research into something new. You’ll often unearth a new subreddit by doing a search for some technique or question. Or you’ll come across a niche web forum where someone asked the same question you have.

And at least half the answers are stress-inducing ‘solutions’ dripping with anxiety. Try again with a different question: same result, this time on another forum.

The regular person starts to ask themself: am I not worried enough? All these parents don’t let their kids go one block away from the house without supervision. All these kids have 5.0 GPAs and are getting Harvard rejections. All these guys are saying I’ll embarrass myself at pickup basketball until I’ve spent 10,000 hours shooting free throws by myself.

Some will turn away. I’m glad the teenage version of myself was able to do that with college applications. But some percentage, possibly a very high one, will get sucked in to the neurosis themselves. Perhaps they’ll sign up, and continue to read content in these forums. Repetition makes truth, and soon they’ll be thinking neurotically, too. They’ll start posting similar answers, to be found by the next unsuspecting web searcher. Repetition makes virulence.

The cycle repeats itself, and the neurosis spreads.

This is a basic memetic effect. The meme is neurotic thinking. Might it be the case that becoming anxious and stressed about one aspect of our lives carries over into others? Perhaps. Take a look at social media recently, and think about whether it’s becoming more or less neurotic. Especially since the election.

Be careful what you’re consuming. And don’t be a keyboard warrior.

Cut through the bullshit: we avoid neurosis here. Join thousands by subscribing for email updates and following me on Twitter, where I also post daily.

Close this section

Elevators in an age of higher towers and bigger cities


After the familiar sound and sight of an elevator door sliding open, I step out into what could loosely be called a room, surrounded by jagged walls of limestone with mossy lichen inching up from the floor and water slowly dripping from the ceiling.

Living walls may be very of the moment for corporate lobbies, but seeing as I’m 350 meters (about 1,148 feet) underground—inside a former limestone mine about an hour outside of Helsinki, Finland—I’m pretty sure this wasn’t the work of an inspired interior designer. The water, I later learn, is from a reservoir a few hundred meters away.

I’m at Tytyri, an underground testing facility run by Kone, one of the big four firms in the multibillion-dollar global elevator market. In mankind’s continued quest to build skyward, this Finnish company figured out that the best way to test elevator technology was to go down, carving out a more cost-effective series of 11 test tracks, or shafts, from an abandoned section of a mine.

The longest test shaft inside this high-rise laboratory can send elevator cabs at speeds up to 90 kilometers (or about 55 miles) an hour. London’s Shard, the tallest skyscraper in western Europe, tops out at 310 meters.

The visitor elevator at Tytyri.

“This is the Area 51 of the industry,” says Tomio Pihkala, the company’s chief technology officer. He’ll later say that the decade-old facility is also the Formula One of the industry, a testing space pushing the limits of hardware, software, engineering, and design.

One of their safety demonstrations—which, thankfully, I don’t participate in—simulates an elevator cab free-fall: Yellow steel blocks, weighing 75 kilograms (165 pounds) each, are stacked in the cabs as test dummies. Brakes deploy and stop a plummeting elevator, which drops with the force of a 10-ton truck hitting a wall at 100 kilometers an hour.

Despite our fascination with skyscrapers—ever-rising buildings, eye-catching profiles, and cutting-edge building technology—elevator technology rarely raises our collective pulse (unless, of course, we get stuck). Roughly 18 billion elevator trips occur annually in the United States, according to the National Elevator Industry Inc., and it’s a safe bet most of those are anything but memorable.

As Pihkala says during my late-March trip to Tytyri, “What’s the big deal about a steel box moving up and down?”

Quite a bit, in fact, if you’re interested in expanding skylines, growing cities, and our race for ever-taller towers. Elevators have been, and will continue to be, central to the story of urbanization and skyscraper architecture.

In 2015, according to Kone, 840,000 new elevators were installed in the world (half in China), adding to the roughly 14 million currently in service worldwide. Kone expects that number to increase sharply as an expected wave of urbanization hits developing countries, especially Nigeria, Indonesia, and India. The United Nations estimates urban populations will grow by 2.5 billion by 2050, with 90 percent of this growth concentrated in Asia and Africa. Kone’s president and CEO, Henrik Ehrnrooth, even went so far as to call elevators “urban mass transit.”

Kone has developed and refined a technology on display at Tytyri called UltraRope. It’s a system to hoist elevators with lightweight, carbon-fiber straps that help designers and architects build higher with less weight. UltraRope will be installed in the forthcoming kilometer-high Jeddah Tower, set to be the world’s tallest building when it’s finished in Saudi Arabia, possibly as soon as sometime next year.

Not every new tower will be record-setting, but with urban growth and increased density, the world is certain to be spending a lot more time in elevators. Kone, like its three big competitors—Otis, ThyssenKrupp, and Schindler—wants to find ways to make your time in these steel boxes more efficient and engaging.

“Our mission is to improve the flow of urban life,” says Ehrnrooth.

During one of my test rides at Tytyri, I get a chance to see what that future may look like. Many aspects of an elevator ride, such as top speeds and safety requirements, are governed by local building codes or regional customs. Kone engineers tell me, for example, that North Americans like a faster run, while Asians prefer a slower acceleration and deceleration. But Kone engineers and designers are exploring how lights, music, and even scents can improve your ride.

Inside one of the illuminated elevators at Tytyri.

Inside a glistening white elevator cab, pressurized to make my brief ride even more comfortable, I’m whisked into the bowels of the earth in what could best be described as a mobile spa waiting room. I feel very out of place dressed in a hard hat and safety boots for the trip underground. Soft ambient music plays as I drop at 6 meters a second, and lights pulse in multiple colors. This is all part of Kone’s efforts to make the time you spend in their products more enjoyable and relaxing.

Like any contemporary company, Kone makes design and innovation a central part of its pitch, talking about user experience and market disruption: In addition to testing systems that allow you to control elevators cabs with a mobile device, they’ve also held hackathons to develop more efficient transportation solutions.

A recent partnership with IBM’s Watson will analyze trip data to see how Kone can better optimize travel between floors. We may dismiss elevator rides as less than remarkable, but for Kone, that seems to be the point: making the experience of navigating increasingly dense urban environments as frictionless as possible.

It’s also about the money. Ehrnrooth says Kone—a firm with 50,000 employees that operates in more than 60 countries—sold 158,000 elevators and escalators last year, adding up to net sales of 8.8 billion euros (about $9.4 billion), or about 20 percent of the global market share. Maintenance of the company’s existing base of 1.1 million devices adds even more income. According to Ehrnrooth’s estimates, the company sells more than one elevator or escalator every minute during the working week, a figure that’s poised to rise.

“Every day, 200,000 people around the world move into cities,” he says. “And in many countries, people are marrying later, forming families later, living alone in the city for a longer period of time and deciding not to move into suburbs.”

In other words, in a century set to be one of growing urban populations, it’s good to be an elevator maker.

During part of the visit, I’m given the chance to stand on top of an elevator cab as it inches up and down, running at what Antti Hoppania, the research and development director, tells me is “maintenance speed.” I can see the UltraRope in front of me—a series of thin black straps, each roughly 1.5 centimeters thick—slowly hoist the elevator, steadied by a series of guardrails lining the test shaft. It’s much sleeker, and 90 percent lighter, than the coiled-steel ropes traditionally used in elevator systems.

UltraRope isn’t a new technology. In 2013, the Marina Bay Sands Casino in Singapore opened with an UltraRope system, and it’s been refined ever since. But by far the most high-profile example of the technology will be the Jeddah Tower, which the company hopes will serve as a massive ad for the new technology.

Jeddah Tower rendering Jeddah Tower rendering
A rendering of the Jeddah Tower.
Adrian Smith + Gordon Gill Architecture

According to Peter A. Weismantle, director of supertall building technology at Adrian Smith + Gordon Gill Architecture, the firm that designed the Jeddah Tower, UltraRope came along at the right time.

The firm, known for its boundary-pushing design, had already stretched the limits of existing technology. The service elevator on the Burj Khalifa—the current tallest building in the world—which runs nearly 1,700 feet tall (one and a half Willis Towers), pushed it even further.

“I have to hand it to Kone, because they saw that we’re in this new age of the supertall building, and there’s a need,” he says. “That’s what pushes the technology.”

Weismantle says that UltraRope, which will be used in the Jeddah Tower’s 59 elevators, helps cut down the size and weight needed for elevators, allows for quicker rides, and makes it easier to build taller structures. In an era of ultrathin towers, such as the designer residential buildings on New York City’s Billionaire’s Row, technology that can save space is welcome.

Other technology on the horizon seeks to make a similar impact on building construction and urban design. One of Kone’s competitors, ThyssenKrupp, is currently erecting a test tower in Rottweil, Germany, to showcase their new MULTI system. A series of elevators cabs that runs on a maglev system, this underdevelopment technology would allow multiple cabs to run in the same elevator shaft, making the task of moving crowds through a building more efficient. The technology even promises to allow for vertical and horizontal travel.

The MULTI System by ThyseenKrupp, which uses maglev technology.
A rendering of how the MULTI system would work within a high-rise building.

Thomas Felis, the vice president of innovation for ThyssenKrupp Elevator Americas, believes the technology can free architects and designers from the big-box model of skyscrapers and towers and lead to more efficient building designs.

“I think we’ll see more unique, efficient city design,” he says. “The use of space in cities will only become more and more important.”

In an age of rapid urbanization, denser blocks and taller buildings will add to the challenge of moving people throughout cities. Weismantle says that plenty of technical challenges stand in the way of building up. The impact of wind on taller and taller structures requires more deft, technical designs that “confuse the wind” and prevent sudden movements. Safety and evacuation become more complex as architects aspire upwards, and supertalls, by their very nature, size, and large footprint on the ground, raise the question of economic feasibility and environmental sustainability.

But one of the most vexing challenges, Weismantle says, is recognizing that it’s not ideal to “isolate human beings in cubes in the sky.” Designers and architects need to find ways to introduce the public space into the higher realm. Perhaps as better, faster, and more efficient elevators help buildings go higher, they can also help make it easier for high-rise dwellers to stay grounded.

Close this section

Show HN: Emojielog – A simple emojie journal application