Tales from Firmware Camp

Tuesday, September 10, 2019 · 7 min read

Last week I attended the Open Source Firmware Conference. It was amazing! The talks, people, and overall feel of the conference really left me feeling inspired and lucky to attend.

Having been pushed to attend vendor conferences and trade shows through my career for various jobs, it was so refreshing to have the chance to hang out with folks from such a genuine community that really just want to help one another.

When the talks hit YouTube you should be sure to check them out (I also tweeted about a few of them). What I will focus on in this post was the last two days of the conference that were devoted to the hackathon.

I had bought a X10SLM-F Supermicro board off of eBay a few months ago that I wanted to run CoreBoot on. If you are interested in finding a board that will work with CoreBoot, you should check its status on the status page. I had been talking to Zaolin about wanting a board to hack on and he recommended this one.

At the hackathon, we decided to start with the BMC instead of the CPU BIOS. This made for some fun problems and definitely a lot of lessons learned. I had a Dediprog SF100 flash programmer I brought to the hackathon as well. Some people use RaspberryPis as their flash programmer but the Dediprog was recommended to me and definitely came in handy. However, if you want a cheaper alternative there are a bunch of ways you can skin that cat.

To get started, we read the original binary off the SPI flash… this worked fairly simply. We used the opensource dpcmd tool from dediprog to do it. But you could also use flashrom.

While inspecting the original binary, we found the string linux a few times… as well as a MAC adress, boot commands, IP address, and some other interesting strings.

Before flashing on new firmware we also made sure the board actually booted the BMC. We didn’t have access to any console so we made due with an IPMI LAN port and dnsmasq to work with DHCP. It worked and we got into the BMC user interface over the web. If you’ve ever used a Supermicro server I probably don’t need to tell you that it’s a piece of shit running a 2.6 linux kernel on the BMC. Getting to the UI proved the board actually booted with the original BMC firmware so we began to break it by trying to run OpenBMC.

Our board has a ASPEED 2400 BMC. We chose a OpenBMC configuration that would give us a kernel supporting that chip. Thanks Joel Stanley for all your work on the kernel patches for all the BMCs. We flashed the SPI flash with our new BMC firmware image and attempted to power on the board.

I am going to interrupt the story here for a second to explain the pain involved with this development cycle of writing to firmware to SPI flash. The SPI flash is 16MB but requires erasing the previous contents (4KB per sector) before you can even write. A delete cycle of a sector is 120ms per sector at the worst. So that’s definitely not ideal and anything you can do to make this faster is very much so ideal. Most flash programmers will not rewrite a sector if its contents have not changed which helps, but still super painful coming from the workflow of a software developer.

Back to our board… our OpenBMC image we flashed didn’t work. Again, a lot of this would have been easier to debug with a serial console but we didn’t have one and we didn’t have the spec to get a UART. Our assumption from this failure was that the IPMI LAN port we were using was not the same port configured for that specific configuration.

So we went to build a custom kernel…

With the help of Joel we built a custom kernel completely separate from OpenBMC, however we flashed the kernel directly to the SPI flash without even u-boot, LOL… obviously this didn’t work.

Then we decided to try something easier and had a hunch a different configuration in the OpenBMC project would have the right port enabled. We built the image for that and flashed it onto the SPI flash. This was arguably faster than making our own OpenBMC configuration with our new kernel.

It also didn’t work, but here we got into a bit more trouble. After this point we could no longer write to the SPI flash. The problem was the BMC was interfacing with the SPI flash and we couldn’t take over the ability to write to it. The SPI flash only allows one device to interact with it at a time. We also could not flash the SPI flash without the board powered on because the entire board was pulling power which was too much for our flash programmer to handle. This is a huge pain in the ass. It turns out it is such a pain in the ass that people have made solutions for it.

Fortunately for us, Felix Held had just given a talk on this pain the day before and he was also in the room. He had one more prototype of his tool, qspimux, and we got to use it on our board.

Qspimux allows for the access to a real SPI flash chip to be multiplexed between the target and a programmer that also controls the multiplexer. This way we could flash the SPI flash with the board powered off.

To get his tool installed we had to de-solder the SPI flash and solder it back on after getting the qspimux parts attached. Props to Edwin Peer for his awesome soldering skills here. Here is a live action shot…

After finishing this, we could write to the SPI flash again. At this point we were trying to re-flash the original Supermicro flash onto the board, just to make sure we didn’t mess anything up along the way. This proved to be more difficult than we thought. We got the firmware to write to the flash but the board still wasn’t working. We verified with the oscilliscope that data was indeed leaving the MOSI (master-out-slave-in) pin and the clock was working on the flash.

Then I tried to read the firmware back from the SPI flash chip to make sure it was indeed our original flash. We suspected that maybe we were writing to the device too quickly. This was indeed the case. The two firmware images did not match. I then wrote the firmware to the SPI flash on the slowest setting just to be sure. Then I could actually verify the image we wrote and the image we read back matched our original firmware image. At this point everything was kosher and we knew the image on the SPI flash chip was indeed the same as the original we pulled off the board the day before.

At this point the board was still not booting the original firmware image. This is when we had to go home and firmware camp was over. Overall, this was a great learning experience. I would have been sad had everything gone smoothly because we would not have learned as much about how to debug all the components of the SPI flash and board. I definitely have not given up on this board and will continue down this rabbit hole until it has open source firmware on the BMC and open source BIOS for the CPU.

I would like to thank everyone at the Open Source Firmware Conference for making this a truly amazing week and specifically those who helped with the crazy hackathon project: Rick Altherr, Edwin Peer, Joel Stanley, Felix Held, Bryan Cantrill, Jacob Yundt (who I can’t seem to find online), Joshua M. Clulow, and everyone else I am forgetting who gave us wires, clips, cords, and whatever else we needed to get this thing going! It truly takes a village.

I cannot wait for the next OSFC, but until then I will work on playing with a logic analyzer to see if what the BMC is reading from the SPI flash is even the right data ;)