What I learned from my failed attempt of writing baremetal android in Rust

This post is focused mostly on the tools that I use while I failed to write a bootable kernel image in rust.

Every year I define a super ambitious goal for my learning process to keep myself motivated on the way. This year I defined my goal as writing a bootable kernel image for my old HTC One X android smartphone. I knew it was going to be hard but I never thought I’d fail in the end. It was clearly the Dunning–Kruger effect that made me think that I can achieve what I want to do with my limited knowledge/experience on the subject.

Prior Work

Let’s start by looking into the projects that have been done to run baremetal code on android smartphones. Unfortunately, I managed to find only two projects out in the wild.

The first project (nexus7-baremetal) made me really excited because I thought nobody would ever care about writing baremetal android and also it was the only resource I had found until I gave up. The project contains some code from raspberrypi/bootloader05. This is because of the shared type of CPU family between Raspberry Pi 2 and Nexus 7 (and HTC One X as well) which happens to be ARM Cortex-A7.

The second project is lktris. The only thing makes this project interesting is it is built on top of littlekernel.

I wanted to try nexus7-baremetal project before I dive into writing my own code in rust but I couldn’t manage to run it successfully even though I told the author of the project the opposite. I thought it would be rude to make him waste his time on a project that he wrote 6 years ago and I wanted to do more research to understand the issue without any hand-holding.

I spent sometime to refresh my knowledge about android-ndk and android-sdk to be able to compile and unsuccessfully run nexus7-baremetal. It’s a bit pain to install standalone android toolchain on macOS and installing platforms, platform tools and emulators is just whole another story that I don’t want to talk about. The below command just shows how badly android sdkmanager cli is designed:

sdkmanager "system-images;android-19;google_apis;armeabi-v7a"

If you really want to use android standalone toolchain on macOS, you can run the following:

# install android standalone toolchain
brew install intel-haxm
brew install android-sdk
brew install android-ndk

# update env vars
export ANDROID_HOME=/usr/local/share/android-sdk
export ANDROID_NDK_HOME=/usr/local/share/android-ndk

# update path
export PATH=$ANDROID_HOME/tools:$PATH
export PATH=$ANDROID_HOME/platform-tools:$PATH

What I learned

Little Kernel

LK (Little Kernel) is a tiny operating system suited for small embedded devices, bootloaders, and other environments where OS primitives like threads, mutexes, and timers are needed. It also initializes the most important hardware such as MMU and UART.

LK is the Android bootloader and is also used in Android Trusted Execution Environment - “Trusty TEE” Operating System.

Android bootloader supports specially packed Android Boot Images only. These files contain the kernel, a ramdisk (root filesystem) and some metadata. The file header of these images includes sizes of all packaged files and the loading address of the kernel.

This header has a size of 0x8000 bytes followed by the kernel image. That’s why the loading address needs to be set to KERNEL_LOADING_ADDRESS - 0x8000 to get LK to the right place.

0x8000

0x8000 (32K) is in fact the size of an offset that leaves space for the parameter block in ARM architecture.

According to the ARM booting procedures:

Despite the ability to place zImage anywhere within memory, convention has it that it is loaded at the base of physical RAM plus an offset of 0x8000 (32K). This leaves space for the parameter block usually placed at offset 0x100, zero page exception vectors and page tables. This convention is very common.

Rust Cross Compilation

It’s a bit complicated to cross-compile rust binaries on macOS for armv7 and you probably knew it already. However, I am ignorant and stubborn and I battled my way to get a proper armv7 toolchain for my macbook. All I wanted to do was just to compile my project to armv7-unknown-linux-gnueabihf platform.

The first thing I’ve done was madly downloading all the packages I’ve found for Homebrew because I didn’t want to deal with crosstool-ng. Nevertheless, I end up installing it and after many failed attempts of building armv7-rpi2-linux-gnueabihf, I realized that macOS is no longer supported by crosstool-ng.

I deciced to do what any sane person would do and fired up a vagrant machine, installed all the toolchains needed and finally, the dysfunctional kernel image was compiled and linked successfully.

Why would I use a VM just to compile a binary? We are in 2019, right? I would have been OK if it was a container but this is a HUGE VM!

I went straight back to the list of Homebrew packages and figured out the only way to compile and link my kernel image is targetting armv7-unknown-linux-musleabihf by installing arm-linux-gnueabihf-binutils. Some would disagree my decision to use musl toolchain considering that baremetal code doesn’t need libc but it was the only viable way for me at that time and if you know a better way (you probably know), please let me know because I don’t have much knowledge about cross-compilation of low-level languages.

Rust targets

There is a list of all the available supported platforms and you can easily add any of them by using rustup.

rustup target add armv7-unknown-linux-musleabihf

To compile your program for a specific target you can either use cargo with --target flag:

cargo build --target=armv7-unknown-linux-musleabihf

or create .cargo/config file:

[build]
target = "armv7-unknown-linux-musleabihf"

cargo-binutils

cargo-binutils is a pretty handy plugin if you need to use LLVM tools for binary inspection and manipulation. It simply proxies the LLVM tools in the llvm-tools-preview rustup component and provides subcommands to invoke any of the tools.

Most of the tools in llvm-tools-preview are LLVM alternatives to GNU binutils. The main advantage of these LLVM tools is that they support all the architectures that the Rust compiler supports.

Rust inline assembly

Currently, there are two feature gated ways to write assembly: nasm! (requires #![feature(asm)]) and global_asm! (requires #![feature(global_asm)]) macros.

asm

nasm! uses the same basic format as GCC uses for its own inline nasm and restricts your inline assembly to fn bodies only. The syntax isn’t the best:

asm!(assembly template
   : output operands
   : input operands
   : clobbers
   : options
   );

The assembly template is the only required parameter and must be a literal string. Here’s an example (taken from the rust book):

#![feature(asm)]

fn foo() {
    unsafe {
        asm!("NOP");
    }
}

fn main() {
    // ...
    foo();
    // ...
}

global_asm

global_asm! gives you ability to write arbitrary assembly without the restriction of fn bodies.

A simple usage looks like this:

global_asm!(include_str!("boot.S"));

.section ".text.boot"

.globl _boot

_boot:
    bl      not_main

.section .text

.globl _put32

_put32:
    str     r1,[r0]
    bx      lr

Using extern Functions to Call Assembly Code

extern keyword facilitates the creation and use of a Foreign Function Interface (FFI). The below example demonstrates how to set up an integration with _put32 function in boot.S.

extern "C" {
    fn _put32(f: &u32, c: &u8);
}

fn main() -> ! {
    unsafe {
        _put32(&0xFF002000, &72);
    }
    loop {}
}

Calling Rust Functions from Assembly Code

extern also has another usage that allows us create an interface for other languages to call Rust functions. You need to add extern keyword and specify the ABI to use just before the fn keyword. We also need to add a #[no_mangle] annotation to tell the Rust compiler not to mangle the name of this function.

In the below example, we make not_main function accessible from boot.S file:

#[no_mangle]
pub unsafe extern "C" fn not_main() -> ! { }

Epilogue

That’s it! I consider this work as a huge win even though I failed to write a functional bootable image. I learnt to use quite useful tools on the way and now I have a better understanding around cross compilation.