This tutorial assumes you have already built and deployed Android 6.0.1 for the Wandboard on an SD-card, according to our preceeding blog post in this series. For quick reference, we are using the Android source 6.0.1 for the Wandboard from the official website. The source code is assumed to be extracted to
Android system development on an SD-card can be quite cumbersome. Even though there is the possibility of pushing and pulling files via Android Debug Bridge (ADB) between the target device and the host computer, more often than not, you still need to replace the contents of the SD-card manually. In the following, we will show you how you can boot the Linux kernel via TFTP and the Android operating system over NFS.
Step 1: Setup tftp-server on the host
The TFTP server will supply the Linux kernel and the device tree files.
# Install tftp server sudo apt-get install tftpd-hpa # Config tftp server sudo vim /etc/default/tftpd-hpa
In our example, we are setting the tftp root directoy to “/srv/tftpboot”, and leave the default user as “tftp”. For reference, our configuration file looks like this:
# /etc/default/tftpd-hpa TFTP_USERNAME="tftp" TFTP_DIRECTORY="/srv/tftpboot" TFTP_ADDRESS="[::]:69" TFTP_OPTIONS="--secure"
Create the tftp root directory and start the server:
# Create tftp root directory sudo mkdir /srv/tftpboot # Change owner of tftp server directory to the tftp user (or whichever user you specifed above) sudo chown -R tftp /srv/tftpboot # Start tftp server sudo service tftpd-hpa start # Check the status of the tftp server sudo service tftpd-hpa status
In the same manner, you can also stop and restart the tftp server. Now copy the kernel and the device tree files to the tftp root directory
# switch to kernel directory cd /opt/android/wand/kernel_imx # copy kernel sudo cp arch/arm/boot/zImage /srv/tftpboot # copy device tree files for Wandboard Quad and/or Dual sudo cp arch/arm/boot/dts/imx6q-wandboard.dtb /srv/tftpboot sudo cp arch/arm/boot/dts/imx6dl-wandboard.dtb /srv/tftpboot
Step 2: Setup nfs share on the host
# install nfs server sudo apt-get install nfs-kernel-server # config nfs server sudo vim /etc/exports
We will use
# /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). # # Example for NFSv2 and NFSv3: # /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check) # # Example for NFSv4: # /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check) # /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check) # /srv/nfsandroid 192.168.1.0/24(rw,no_root_squash,sync,crossmnt,no_subtree_check) 127.0.0.1/24(rw,no_root_squash,sync,crossmnt,no_subtree_check)
Now create the nfs share directory, export the NFS table, and start the server:
# create nfs share directory sudo mkdir /srv/nfsandroid # create nfs table sudo exportfs -a # start nfs server sudo service nfs-kernel-server start
We need to populate our nfs share with the Android operating system. There are multiple ways to achieve this. The approach chosen here is to extract the ramdisk (containing the root file system) to the nfs share, and then mount the Android system image inside of it. By the way, if you were to merely copy the contents of the root directory from the out directory, the system will not boot beyond init, because the permissions of the *.rc files are not correct yet.
user@host:~$ cd /srv/nfsandroid user@host:/srv/nfsandroid$ gunzip -c ~/opt/android/wand/out/target/product/wandboard/ramdisk.img | sudo cpio -i user@host:/srv/nfsandroid$ sudo mount /opt/android/wand/out/target/product/wandboard/system_raw.img system
If you want to unmount system_raw.img again and get a “device is busy” error, you need to stop the NFS server beforehand with:
sudo service nfs-kernel-server stop
After setting up the bootloader environment in the next step, this should already result in a semi-successful boot. Further changes needed are detailed in Step 4.
Step 3: Configure u-boot bootloader arguments to boot from tftp/nfs server
On the Wandboard, we need an SD card with a u-boot bootloader. Remember, the SPL must to be copied at 1k offset into the SD card, and the u-boot.img placed onto the first FAT32 partition.
cd /opt/android/wand # use the command 'lsblk' to determine the device name of the sdcard # and replace /dev/sdX accordingly sudo dd if=./bootable/bootloader/uboot-imx/SPL of=/dev/sdX bs=1k seek=1 sync
Also copy the u-boot.img file to the first FAT32 partition (named BOOT in our example)
cp ./bootable/bootloader/uboot-imx/u-boot.img /media/$USER/BOOT
Startup the Wandboard with the SD-card, and in a serial console, hit enter to stop u-boot from auto-booting.
We now need to change a few u-boot environment variables. You can inspect the current variables with the “printenv” command. Note that there is already an environment variable “netboot”, along with “netargs” with the following contents:
netboot=
echo Booting from net ...;
run netargs;
if test ${ip_dyn} = yes; then
setenv get_cmd dhcp;
else
setenv get_cmd tftp;
fi;
${get_cmd} ${image};
if test ${boot_fdt} = yes || test ${boot_fdt} = try; then
if ${get_cmd} ${fdt_addr} ${fdtfile}; then
bootz ${loadaddr} - ${fdt_addr};
else
if test ${boot_fdt} = try; then
bootz;
else
echo WARN: Cannot load the DT;
fi;
fi;
else
bootz;
fi;
netargs=
setenv bootargs console=${console},${baudrate}
root=/dev/nfs ip=dhcp
nfsroot=${serverip}:${nfsroot},v3,tcp
If we run the “netboot” command, it will first set the boot arguments by running netargs. Depending on the variable “ip_dyn”, the get command for the kernel image is either set to “dhcp” or “tftp”, after which the kernel image is pulled, along with the device tree file. Finally, the zImage is booted with the bootz command. Easy, right? 🙂
As mentioned, we need to set a few environment variables to boot from our nfs server. I’ll show all the needed commands, and explain them later on:
=> setenv ipaddr 192.168.1.188
=> setenv serverip 192.168.1.163
=> setenv nfsroot /srv/nfsandroid
=> setenv ip_dyn no
=> setenv bootfile zImage
=> setenv fdtfile imx6q-wandboard.dtb
=> setenv netargs 'setenv bootargs ${bootargs_base} console=${console},${baudrate} root=/dev/nfs rw ip=dhcp nfsroot=${serverip}:${nfsroot},v3,tcp'
=> env save
The variable “ipaddr” sets the IP address of the target, and must be set to an IP address for the tftp command. If you don’t know which IP address to use, find out with the dhcp command:
=> setenv autoload no => dhcp BOOTP broadcast 1 DHCP client bound to address 192.168.1.188 (4ms)
Here, the variable “autoload” is temporarily set to no, so that dhcp does not automatically try to load a kernel image when it is called. You may wonder why even use tftp instead of the dhcp command for downloading our kernel image? The reason for this is that – at least for me – every time the dhcp command is used, the serverip variable is reset to 192.168.1.1. So by setting “ip_dyn” to to “no”, we choose the tftp command for downloading the kernel and device tree files, which seems to work better here. (It’s very well possible that dhcp would work somehow too. In either case, tftp is fine for now)
The variable “serverip” is set to our host’s IP address. It is also read by tftp, see the official U-Boot wiki section on U-Boot Environment Variables. As mentioned, “ip_dyn” is set to “no” so that tftp instead of dhcp is used. “bootfile” is the default image to load for tftp (again, check the wiki section for more information). “fdtfile” is the name of our device tree binary on the tftp server.
We are also slightly altering “netargs” to mount the root directory as read-write (rw) and by adding ${bootargs_base}, which contains the init location, along with video arguments, etc. (This was probably just forgotten in the default settings).
Needless to say, adapt “ipaddr” and “serverip” accordingly, then save the environment.
=> env save Saving Environment to MMC... Writing to MMC(0)... done
If something went wrong, you may reset to the default values with “env default -a”. Otherwise, you can now manually boot via network by running netboot:
=> run netboot
Furthermore, if you inspect the “bootcmd” variable – which is run by default when the device boots up – you’ll notice that among other things it will try to find and load a ramdisk and kernel image from the the first FAT partition. However, if this fails, it will resort to “run netboot” in the end. So in case you still have a zImage and uramdisk.img on the SD card, you can simply delete or rename those files, after which it will always boot from TFTP/NFS upon rebooting.
Step 4: Changes to Android to fix NFS booting
As mentioned, we should already have a somewhat successful boot. At least, we need to adapt the fstab.freescale file. When booting from SD-card, the system folder is mounted from a separate partition. However, when booting via NFS with the above setup, we already have the system folder inside our NFS root directory. So delete or comment out the line for mounting the system
# Android fstab file. #<src> <mnt_point> <type> <mnt_flags and options> <fs_mgr_flags> # The filesystem that contains the filesystem checker binary (typically /system) cannot # specify MF_CHECK, and must come before any filesystems that do specify MF_CHECK /devices/soc0/soc.0/2100000.aips-bus/2190000.usdhc/mmc_host* auto auto defaults voldmanaged=sdcard:auto /devices/soc0/soc.0/2100000.aips-bus/2184200.usb/ci_hdrc.1* auto auto defaults voldmanaged=usb:auto /devices/soc0/soc.0/2200000.sata/ata1* auto auto defaults voldmanaged=sata:auto #/dev/block/mmcblk2p2 /system ext4 rw,barrier=1 wait,verify /dev/block/mmcblk2p4 /data ext4 nosuid,nodev,nodiratime,noatime,nomblk_io_submit,noauto_da_alloc,errors=panic wait,check /dev/block/mmcblk2p3 /cache ext4 nosuid,nodev,nomblk_io_submit wait,check /dev/block/zram0 none swap defaults zramsize=314572800
Of course, this also means that we should have an SD-card with data and cache partition inserted. You may need to change the partition numbers p4 and p3 for your purpose. The above fstab file should be OK if you have set up your SD card as described in this previous post. For completeness’ sake, here is what is assumed to be on your SD card:
- 1MB offset or more before first partition.
The u-boot SPL is placed here at an offset of 1024 bytes. - Partition 1: BOOT, VFAT, ~32MB
u-boot.img should be copied here, and will be loaded by SPL. - Partition 2: SYSTEM, EXT4, ~1024MB
This holds the system partition, and is not loaded with the above fstab file. - Partition 3: CACHE, EXT4, ~1024MB
Cache partition - Partition 4: DATA, EXT4, remaining size
Data partition
Now we can try to boot again. You should see a fair amount of log output, but after a short time, the system will hang with the error message
... nfs: server 192.168.1.66 not responding, still trying nfs: server 192.168.1.66 not responding, still trying nfs: server 192.168.1.66 not responding, still trying nfs: server 192.168.1.66 not responding, timed out
Finding out what exactly goes wrong here is kind of hard, and is a great chicken-and-egg problem if I have ever seen one. Obviously, there is something wrong with the network. However, as soon as we see the error message, it’s already too late for debugging since our system with all debugging tools is only available via network. As it turns out in the end, the error stems from the “netd” service. The corresponding source code is found inside the Android source under
When inspecting CommandListener.cpp, you can already see that something was patched here to read a system property “ro.nfs.mode”. This actually comes from a Freescale Android KK443 (KitKat) patch release. Setting this property does not fix our issue just yet, but certainly points us in the right direction. After a lot of trial & error, it turns out the problem lies within RouteController::Init function, specifically by calling flushRules() which will clear any present routing rules and in turn break our ethernet connection to the NFS server. We could simply modify the flushRules() function inside RouteController.cpp, but then we would need two different versions of netd – one for normal and one for NFS booting. Therefore, let us also use the “ro.nfs.mode” property to determine whether we should call flushRules(). A possible fix for RouteController.cpp could look like this (note that you also need to include cutils/properties.h)
// NFS fix for system/netd/server/RouteController.cpp
#include <cutils/properties.h>
...
static bool isNFSbooted() {
char prop_value[PROPERTY_VALUE_MAX] = {'\0'};
if (property_get("ro.nfs.mode", prop_value, "no")) {
if (strcmp(prop_value, "yes") == 0)
return true;
}
return false;
}
int RouteController::Init(unsigned localNetId) {
if (!isNFSbooted()){
if (int ret = flushRules()) {
return ret;
}
}
if (int ret = addLegacyRouteRules()) {
return ret;
}
...
Re-compile netd and copy the new binary to the NFS directory:
# If not done already, initialize build $ cd /opt/android/wand $ source build/envsetup.sh $ lunch wandboard-eng # rebuild netd $ mmm -j4 system/netd # copy netd to NFS directory $ sudo cp out/target/product/wandboard/system/bin/netd /srv/nfsandroid/system/bin/netd
When booting via NFS we also need to add the mentioned system property in init.freescale.rc’s on boot section.
on boot
# Add NFS property to prevent ip rules flushing
setprop ro.nfs.mode yes
# emmulate battery property
setprop sys.emulated.battery 1
...
That should do it, reboot und gut 😀
(There are also some more patches in the Freescale KitKat release. This one disables clearing of the interface addresses from within framework when in NFS mode. In Android 6, you would need to add those changes to EthernetNetworkFactory.java inside frameworks/opt/net/ethernet, however it does not seem this patch is needed here. Another patch is for installd, but again, everything seems to work fine here without)
Bonus: Where does u-boot save environment variables on the SD card?
If you remember, we saved our u-boot environment variables to the SD card. But where exactly are those variables stored now? Have a look at the Wandboard config file to find out:
vim bootable/bootloader/uboot-imx/include/configs/wandboard.h
Search for “CONFIG_ENV” and you will find the following section:
/* FLASH and environment organization */ #define CONFIG_SYS_NO_FLASH #define CONFIG_ENV_SIZE (8 * 1024) #define CONFIG_ENV_IS_IN_MMC #define CONFIG_ENV_OFFSET (6 * 64 * 1024) #define CONFIG_SYS_MMC_ENV_DEV 0
So the environment is 8*1024 bytes = 8 KB in size. It resides in the MMC (or SD card), at an offset of 6*64*1024 bytes = 384 KB. You can double check that by saving your own variables, and then dumping the environment from the SD card with the following command:
sudo dd if=/dev/sdX of=./uboot.env bs=1024 skip=384 count=8
You can then check the contents of uboot.env with a hex editor such as “ghex”.