First blog here we go!

So firstly some introduction I suppose. I am Shreyansh, a Summer of Bitcoin 2022 mentee, working on implementing multi-peer neutrino and more for Bitcoin-S's neutrino node.

This post would be more like a report on everything that I have done till now, written retrospectively though I do hope to write stuff as it happens from now on.

Anyways, I would focus mainly on the major milestones and interesting info in hope that you, the reader, atleast makes it to the end. In chronological order, first was

Running a Neutrino Node written in Scala for desktop environments in Android!

This was something my mentor, Chris, wanted me to look into so I gave it a go. So there's a lot going here on the problems with doing this so I'll break it down. The big idea was that since Android can run Java and Scala compiles to Java, so can we run our Neutrino Node that was written for desktop environments like Mac, Win and Linux, on Android. Turns out you can, as I was ultimately able to do, but there are a LOT of catches.

Firstly, Android runs Java but it does not exactly use the usual Java and jvm as on Linux (or Win/Mac, in the current context, Linux means desktop). Android's Java is a subset of the Java we compiled and ran against. There's no JVM on Android, instead you get ART (Android Runtime Environment). When android creates an apk it "dexes" the java files which is basically a fancy term for converting java to bytecode such that it is ART compatible. Note that not all java code can be converted. For writing apps that's not problem at all, since you would be writing against the Java subset that ART uses which is provided with Android SDK. For all other cases, as was ours, it's not so easy. And if you use something that is not part of the Android's Java, well, things get messy.

The aim here is for me to the Neutrino Node syncing on an android device. First thing that I did was get all the jar artifacts related to node and try to get it compiling. Working or not we can figure that out later :). First issue was the most mundane one, a packaging issue. So android has a REALLY (and I can't emphasize this more) REALLY big issue which is that, whatever code be it your own or from a dependency, it places it all in the same path. I may not clear here so let me give some context on apks.

So unlike Windows' .exe or Linux's binaries, apks are just a zip archive of a folder containing all the compiled files and resources. For code files it makes sense, because if actually cannot have two files with the same path, that would be an issue for importing too, it won't even work on Linux let alone android. But resources and files are a different thing altogether. So suppose I have two modules and both have a resource file named application.config, both of them use that file for their own specific tasks. That's no problem, unless you are going to build a fat binary which is what apks are. Android would ignore that they are in different modules and then merge them based on paths inside module, to put it simply, it merges all of them into one module, the problem being, now how would the two different files that have same path be managed. That's the packaging issue. For my case it was particularly severe, so this application.config is actually one of the files that let to this confict. And it was not just two or three of them, bitcoin-s had multiple and Akka, one our dependencies had multiple of them too. What I ultimately did is create a custom config that would be compatible with the akka ones and for node sync. A bit of a hack but the other option would be using a unique name for all of those files which is equally hacky.

Next up was native libraries. Bitcoin-S had some native dependencies too, how would that work with android??? So android too has support for native code execution, as in if you give binaries compatible with the android's architecture, which is usually arm64 these days, you can run those binaries. What binaries we needed? Well there were a few but the most important one was drivers for SQL, apart from some cryptographic libraries. This part was easier but buckle up, we are just getting started.

Getting database operations to work on android, now this was a real challenge. So this is where we significantly see the effect of the divergence between Java and Android's Java. So firstly, SO gods say that using jdbc drivers on android is not recommended simply because there is not much support for this, this union was not meant to happen, as you will understand soon. For basic operations Android's Java does provide it's own database related functions, but we already have all our database code, we don't want to port it! So off I go to try getting sqlite drivers for android. There are two main ones available at the moment, SQLDroid and Xerial, with the latter one being better because the first one did not really support a lot of SQL operations as was evident in my tests too. Xerial was the only choice.

Now xerial did have android support merged in quite recently but never have they made a release with that nor do they have up to date artifacts. Well, off to building from source. Great, but on running I found an exception with SQLDroid driver not found. Huh? well digging a little bit into flyway, our migration management tool, I found that flyway changes the driver type to SQLDroid. ITS HARDCODED "sqldroid". Imagine the horror. Why isn't there an issue fixing this? Oh there is! But that was even more bad news, turns out it was closed with a comment that the latest release removes android support (We weren't using latest flyway at the time). Anyways, I could fix it right. So off we go to building a custom flyway build. And we get a new error.

Turns out you can't use connection pools on android. You get what I mean when I said android and jdbc is bad pair, right? Fixing that was easy but next was Slick, our ORM tool, too used a beans class that was not with Android's Java, this would have been very very hard to deal with but fortunately I found a drop in replacement package for it, that just worked, by the way this was a drop in for Slick, so that meant using a custom Slick build too.

Should be the end of it right? No. Another issue with flyway, apparently it can't find the migration files. Once again I find an issue referencing this but alas, that too was closed unfixed. Thankfully someone had done some work on it and adding his changes to my own fork, I finally got it to work.

With this the node works on android! The performance took a huge hit but it's atleast something. In any case, seems like you are still here. Not sure how you endured these walls of text, but if you (hopefully) did not skip to the bottom, then I guess I was successful in my endeavour of capturing your attention enough to have you make it till here. I initially intended this one blog post to contain all that I did but I guess at this point we should go for a PART 2.

PS: I was also able to get a basic android app written in Scala (YES!) to run on android. No relying on the fact that bytecode is same. You write directly in Scala! Maybe this will be another post too.