So, soru thought she’d write this up, as some might enjoy reading how programs/things get made for embedded platforms such as the Gamebuino META? Well, if you don’t like reading such things, feel free to just skip this entire post ^^
So, first off, how did this all start out? Sylvain asked around in secret a bit who’d be up for bringing python to the gamebuino. Naturally soru had to say that she’s interested. And so it started.
Soru had already previously heard of MicroPython, a python port designed for embedded platform to “fit and run within just 256k of code space and 16k or RAM”. Sounds perfect for the gamebuino, right? So, let’s see if someone already made a port for the Arduino Zero. Why that arduino? Its hardware from a chipset perspective is the same as the gamebuino meta, and it is pretty popular, so there bound to be someone who ported it! And yes, as it turns out adafruit forked MicroPython, creating CircuitPython, and, look and behold, it already has official ports to atmel-samd based bored - including the Arduino Zero!
So…here’s the plan: Try circuitpython on the Arduino Zero (soru has one!), see how it works, try to self-compile it for the zero and then try to get it running on the gamebuino. After that figure out how to add custom library stuff so that the gamebuino library can be accessed from within python. Sounds simple enough, right? Well, then let’s get going!
First steps: trying out circuitpython and the zero…this works splendit! Just uploading the binary they distribute onto the zero, plug it into the computer and it shows up as a flash drive! Great! Drag&drop a python script to make an LED blink in there and BOOM, it just works. Fantastic! So, now let’s take a look at the repo: makefiles seem to work fine, it all compiles, and the resulting binary also works fine on the zero. Awesome! So, only thing left is to port this to the gamebuino, right?
Well, let’s first test that the hardware stuff is really all ok: flash the zero’s bootloader onto the gamebuino, and test the arduino zero circuitpython port on there. Aaaaand, as expected, it works just fine! So, let’s put back the gamebuino bootloader and start porting circuitpython, shouldn’t be too hard, right? Or so soru thought. See, the zero bootloader is
0x2000 in size, while the gamebuino bootloader is
0x4000 in size (due to added sd card game switching capabilities). So…let’s change the bootloader size in the linkerscript, try to compile and…not enough flash available. Damnit! This is where the rabbit hole starts.
As soru mentioned above, circuitpython presents itself to the desktop PC as a flash device and you drag&drop files in there that then get run automatically. So, it has a virtual filesystem within flash itself. What if soru just decreases the size of that a bit so that circuitpython itself has enough flash even though of the larger bootloader? And soru would somehow have to make space for the gamebuino library anyways.
So, let’s just decrease the internal filesystem size! How perfect, ther is a
#define switch for board-specific ports to change that! So, let’s change it, and…it compiles! Uploading to the gamebuino…running it and…nothing. Right, soru forgot to flash the gamebuino bootloader on again. So, let’s flash that and re-do the whole thing, and…it uploads and then…nothing. This meant either a busy wait somewhere or a Hard Fault or something (due to not being compiled with the entire gamebuino library it does not flash the loader with the message “Hard Fault”, it just hangs up). So…time to get out a hardware debugger and see what is going on!
Tracking this problem down was rather difficult, due to its apparent randomness. Eventually soru tracked it down to apparently coming from within micropythons garbage collector - or so she thought. She tried desperately to figure out what exactly it was, but she seemed to be walking around in the blue. So, back to the zero to try if it is reproducable there! Compile circuitpython with bootloader size
0x2000 however, even though unneeded, a smaller filesystem size. And…yep, that just did nothing, too, hanging up. That hinted to soru that, somehow, the
#define switch to change the filesystem wasn’t working right, so…time to read through tons of code! And indeed, as it turns out, the filesystem size was hard-coded to be
0x00010000 all over the place - even if you changed it to something else via the
#define switch provided. Kinda…stupid. This means we are back on track, though! Why did it appear to be in the garbage collector, though? Due to the different actual filesystem size and expected filesystem size the code probably re-wrote itself in the upper addressable space, causing all kind of undefined behaviour.
Tracking down tons of locations where the filesystem size is set and making it actually configurable, let’s compile the sketch, upload it to the zero and…BOOM! There we go! Well, almost. The board started up, however, it didn’t present itself to the computer as a flash drive. How did soru notice it started up, then? Circuitpython also offers a REPL via serial interface, and that one worked just fine! So, compile this again with the gamebuino bootloader size, upload it there and…well, same. REPL works, but not the USB stuff. On the upside that means that python IS already running on the gamebuino, though!
Together with some help on github soru tried figuring out what was wrong: As she had REPL access, she could easily check if the virtual filesystem was detected properly! And, as it turns out, it was not. This hinted strongly that creating it failed somehow. So…time to grab a hardware debugger again, set a breakpoint in
mkfs and power this thing up! After carefully stepping through all the steps and seeing nothing off, soru finally found the issue: It apepared that
mkfs only works if there are at least 50 blocks to create the filesystem in. What does this mean? Well, a filesystem is separated into blocks, for Fat32 this is often 512-byte blocks. So 50 blocks would mean
50*512 bytes. As it turns out, for testing soru made the virtual filesystem a tad too small. So, increase the size a little so that it is more than 50 blocks large (as an alternative, soru could have also decreased the blocksize), compile, try it and…BOOM! The gamebuino showed up as a flash drive, drag&drop the LED script in there and the LED started blinking! Using python! Running on the gamebuino! That means the hard part of porting should be done, right?
Well, or so soru thought. That was only half of the hard work, as it turns out. The idea was to take the existing gamebuino library, compile it into circuitpython and expose methods to python so that you could easily program it. Well…the problem here is that circuitpython is written in C only and the gamebuino library is C++ (with arduino framework). Circuitpython being c only means that its makefiles didn’t handle
cpp files at all, so soru started looking around if there were existing solutions to mix c++ into circuitpython/micropython. And indeed there is right here! Soru tried it out a bit but…it seemed needlessly complex. So she sat down and edited circuitpythons makefiles to include c++ support. That was surprisingly easy! A simple dummy module that loads some simple dummy methods off of a c++ file (that has been compiled with a c++ compiler) and it works. Great first signs!
So, now let’s just add in the gamebuino library. First thinking that maybe soru doesn’t even have to remove the arduino framework stuff and it compiles just fine in, she went through the library and changed
bool and other small things. Well…at the end it didn’t compile well, there were some naming conflicts from within the core arduino includes and some other non-implemented functions. So…let’s just make the gamebuino library independant of the arduino library via a
#define switch. There was nothing too exciting here, this was pretty straight forward. But…well, in the end, while stuff compiled, it didn’t link together. Missing reference to like
__kill and stuff.
Searching some online it appears that those are OS-related methods, and many people just added some dummy methods. Soru tried that, too, but…somehow it still couldn’t find the references. Huh, that is odd. Even with
-lnosys. Looking some more into it it appeared that such linking errors could appear with usage of
free. So…let’s re-implement them! well, provide them and they just link back to micropythons implementation of them (in the garbage collector). Perfect, let’s just compile an-…the link errors still appear. Oh, right, this is C++! It has the keywords
delete, which are also used in the gamebuino library! Those use internally
free, so…time to override them and…yes! It finally links together! Upload the simple program to the gamebuino, connect to it via serial to the REPL, import the test module, run
gb.begin and…hard fault. This time it actually loaded back the loader as the gamebuino library was compiled in.
Nothing to worry about, right? Let’s just grab the hardware debugger and make it break into the hardware debugger instead of flashing the loader, and then examine the stack trace. Shouldn’t be too hard, right? So, let’s just do this and- corrupted stack trace. OK, this can happen sometimes, so instead let’s set a breakpoint at the start of
gb.begin() and slowly step through the code. It seems
gb.begin hard-faulted on
display.fill(Color::black), which is after tft init. Huh, that is…odd. And the even odder thing, is, that when internally calling
_fill it seems to hard fault. That is when it was supposed to call
Image::_fill instead of
Graphics::_fill due to the virtual override. This didn’t make much sense why it would hard fault there, or so soru thought. For testing, she tried another method exposed to python, something along the lines of
Gamebuino_Meta::Gamebuino gb2; gb2.begin(); and, guess what? That one worked just fine, no hard fault at all! So, soru was starting to track this down, great!
So, the symptoms were that the globally defined
gb didn’t work, while if you locally define a
gb2 it works fine. That’s…odd. Soru first blamed the C++/C mixing. Perhaps she had to invoke some C++ init stuff to create variables in the global namespace correctly? But then it hit her - the internal creation of the
gb variable created also things like
gb.display, which is an Image object, inheriting from Graphics. So it needed a vtable for routing the calls correctly. Soru doesn’t know exactly where it is, but she guesses it is somewhere in ram. And, well, here is the problem: On startup the
gb object is created and initialized, and after that when we run
gb.begin() in the REPL it hard-faults. Well, the gamebuino library now uses micropythons
free, so it would kinda make sense if we can only use them properly after micropython started up. So…let’s add a
#define switch to the library to not auto-create the object, and, on usage, automatically create and init the object and store it in a pointer and use that. And, well, it works! Soru can now just run
gb.begin() from inside the REPL!
So, let’s just, for testing purposes, stick this onto a python file and run it via the file drag&drop thing and- hard fault. Wut? Okay, as it turns out sorus approach didn’t properly clean up stuff: it created the gamebuino object on first usage based on the pointer being
nullptr, however, due to switching between REPL and SD interface the python runtime restarted, thus the old garbage collector pointers became invalid. This also meant that it hard-faulted when changing your python file and re-uploading it to the gamebuino. So, soru had to add that, on unloading micropython, it also destroys the gamebuino object. There, perfect, NOW it worked just fine!
The remaining stuff from here was now pretty straight-forward: add bindings for things like tft sending / buttons receiving etc. (as
SPI.h is part of the arduino framework), and bind these methods over to python. This was all pretty straight forward, while there were some hickups along the road (for example, neopixel timings seemed to have to be done differently on circuitpython than “normal” arduino, due to the NVMC being in a different mode). But hey, it’s all working now! Great!
Total time spent to reach this point: about 30 hours.
Soru hopes you enjoyed reading up on this little dev blog, if you have any questions, feel free to ask below!