Porting Apache NuttX RTOS to Avaota-A1 SBC (Allwinner A527 SoC)

📝 16 Mar 2025

Avaota-A1 SBC with SDWire MicroSD Multiplexer and Smart Power Plug

(Watch the Demo on YouTube)

This article explains how we ported NuttX from QEMU Arm64 Kernel Build to PINE64 Yuzuki Avaota-A1 SBC based on Allwinner A527 SoC … Completed within 24 Hours!

Why are we doing this?

We’re ready for volunteers to build NuttX Drivers for Avaota-A1 / Allwinner A527 (GPIO, SPI, I2C, MIPI CSI / DSI, Ethernet, WiFi, …) Please lemme know, maybe we can draw something on the Onboard LCD! 🙏

(BTW I bought all the hardware covered in this article. Nope, nothing was sponsored: Avaota-A1, SDWire, IKEA TRETAKT)

Avaota-A1 SBC connected to USB UART

§1 Boot Linux on our SBC

Nifty Trick for Booting NuttX on Any Arm64 SBC (RISC-V too)

To begin, we observe our SBC and its Natural Behaviour… How does it Boot Linux?

  1. Connect a USB UART Dongle (CH340 or CP2102) to the UART0 Port (pic above)

    Avaota-A1USB UARTColour
    GND (Pin 6)GNDYellow
    TX (Pin 8)RXOrange
    RX (Pin 10)TXRed

    UART0 Port of Avaota-A1 SBC

  2. Boot Log will appear at /dev/ttyUSB0

    ## Allow the user to access the USB UART port
    ## Logout and login to refresh the permissions
    sudo usermod -a -G dialout $USER
    logout
    
    ## Connect to USB UART Console
    screen /dev/ttyUSB0 115200
  3. Download the Latest AvaotaOS Release (Ubuntu Noble GNOME) and uncompress it…

    wget https://github.com/AvaotaSBC/AvaotaOS/releases/download/0.3.0.4/AvaotaOS-0.3.0.4-noble-gnome-arm64-avaota-a1.img.xz
    xz -d AvaotaOS-0.3.0.4-noble-gnome-arm64-avaota-a1.img.xz
  4. Write the .img file to a MicroSD with Balena Etcher.

  5. Insert the MicroSD into our SBC and Boot AvaotaOS. We’ll see the Boot Log…

    read /Image addr=40800000
    Kernel addr: 0x40800000
    BL31: v2.5(debug):9241004a9
    sunxi-arisc driver is starting
    ERROR: Error initializing runtime service opteed_fast
  6. Aha! Kernel Boot Address 0x4080_0000 is super important, we’ll use it in a while

§2 NuttX Kernel Build for Arm64 QEMU

Follow these steps to Build and Run NuttX for Arm64 QEMU (Kernel Build)

## Build NuttX Kernel (NuttX Kernel Build)
git clone https://github.com/apache/nuttx
git clone https://github.com/apache/nuttx-apps apps
cd nuttx
tools/configure.sh qemu-armv8a:knsh
make -j

## Build NuttX Apps (NuttX Kernel Build)
make -j export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make -j import
popd

## Boot NuttX on QEMU
qemu-system-aarch64 \
  -semihosting \
  -cpu cortex-a53 \
  -nographic \
  -machine virt,virtualization=on,gic-version=3 \
  -net none \
  -chardev stdio,id=con,mux=on \
  -serial chardev:con \
  -mon chardev=con,mode=readline \
  -kernel ./nuttx

Check that it works…

- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

NuttShell (NSH) NuttX-12.8.0
nsh> uname -a
nxposix_spawn_exec: ERROR: exec failed: 2
NuttX 12.8.0 96eb5e7819 Mar 13 2025 15:45:11 arm64 qemu-armv8a

nsh> hello
Hello, World!!

## No worries about `nxposix_spawn_exec`
## To Quit: Press Ctrl-a then x

We’re ready to boot nuttx.bin on our SBC.

NuttX Kernel Build will call out to HostFS Semihosting

What’s this semihosting business in QEMU?

## Boot NuttX on QEMU, needs Semihosting
qemu-system-aarch64 \
  -semihosting ...

NuttX Kernel Build will call out to HostFS Semihosting (pic above) to access NSH Shell and NuttX Apps. We’ll change this for our SBC.

Why start with NuttX Kernel Build? Not NuttX Flat Build?

Our SBC is a mighty monster with Eight Arm64 Cores and plenty of RAM (2 GB). It makes more sense to boot NuttX Kernel Build and run lots of cool powerful NuttX App, thanks to Virtual Memory.

(NuttX Flat Build was created for Simpler Microcontrollers with Limited RAM)

Yuzuki Avaota-A1 SBC with PinePhone MicroSD Extender

§3 Boot NuttX on our SBC

Remember the MicroSD we downloaded? Inside the MicroSD is a 28 MB Linux Kernel, named “Image”…

$ ls -l /media/$USER/boot
   78769  bl31.bin
  180233  config-5.15.154-ga464bc4feaff
     512  dtb
     512  extlinux
27783176  Image
  180228  scp.bin
   12960  splash.bin
 5193581  System.map-5.15.154-ga464bc4feaff
 6497300  uInitrd

We replace it with NuttX…

  1. Take the NuttX Kernel nuttx.bin from the previous section

    (Yes the QEMU one)

  2. Overwrite the Image file by nuttx.bin

    ## Backup and overwrite `Image` on MicroSD
    mv \
      /media/$USER/boot/Image \
      /media/$USER/boot/Image.old
    cp \
      nuttx.bin \
      /media/$USER/boot/Image
    
    ## `Image` should be a lot smaller now
    ## Remember to Unmount and prevent filesystem corruption
    ls -l /media/$USER/boot/Image
    umount /media/$USER/boot
  3. Insert the MicroSD into our SBC. Boot it…

    read /Image addr=40800000
    Kernel addr: 0x40800000
    BL31: v2.5(debug):9241004a9
    sunxi-arisc driver is starting
    ERROR: Error initializing runtime service opteed_fast

Nothing happens. We tweak this iteratively, in tiny steps…

§4 Print to UART in Arm64 Assembly

Is NuttX actually booting on our SBC?

Let’s print something. UART0 Base Address is here…

A523 User ManualPage 1839
ModuleBase Address
UART00x0250_0000

16550 Transmit Register is at Offset 0

A523 User ManualPage 1839
OffsetRegister
0x0000UART_THR (Transmit Holding Register)
0x0004UART_DLH (Divisor Latch High Register)
0x0008UART_IIR (Interrupt Identity Register)
0x000CUART_LCR (Line Control)

This says we can Print to UART like so…

// Print `123` to UART0
*(volatile uint8_t *) 0x02500000 = '1';
*(volatile uint8_t *) 0x02500000 = '2';
*(volatile uint8_t *) 0x02500000 = '3';

But we’ll do it in Arm64 Assembly: arm64_head.S

/* Bootloader starts NuttX here */
__start:
  add x13, x18, #0x16 /* "MZ": Magic Number for Linux Kernel Header */
  b   real_start      /* Jump to Executable Code      */
  ...                 /* Omitted: Linux Kernel Header */

/* Executable Code begins here */
/* We print `123` to UART0     */
real_start:

  /* Load UART0 Base Address into Register X15 */
  mov  x15, #0x02500000

  /* Load character `1` into Register W16 */
  mov  w16, #0x31

  /* Store the lower byte from Register W16 (`1`) to UART0 Base Address */
  strb w16, [x15]

  /* Load and Store the lower byte from Register W16 (`2`) to UART0 Base Address */
  mov  w16, #0x32
  strb w16, [x15]

  /* Load and Store the lower byte from Register W16 (`3`) to UART0 Base Address */
  mov  w16, #0x33
  strb w16, [x15]

(RISC-V? Same same)

Rebuild NuttX and recopy nuttx.bin to MicroSD, overwriting the Image file. NuttX will boot and print 123! 🎉

read /Image addr=40800000
Kernel addr: 0x40800000
BL31: v2.5(debug):9241004a9
sunxi-arisc driver is starting
ERROR: Error initializing runtime service opteed_fast
123

Indeed NuttX is booting on our SBC, then crashing later. (Ignore the error: opteed_fast)

Why print in Arm64 Assembly? Why not C?

  1. Arm64 Assembly is the very first thing that boots when Bootloader starts NuttX

  2. This happens before anything complicated and crash-prone begins: UART Driver, Memory Management, Task Scheduler, …

  3. The Arm64 Assembly above is Address-Independent Code: It will execute at Any Arm64 Address

Next we move our code and make it Address-Dependent…

§5 Set the Start Address

NuttX boots a tiny bit on our SBC. Where’s the rest?

Our SBC boots NuttX at a different address from QEMU. We set the Start Address inside NuttX…

read /Image addr=40800000
Kernel addr: 0x40800000
123
  1. Remember the Boot Log from earlier? It says that the SyterKit Bootloader starts NuttX at Address 0x4080_0000. We set it here: ld-kernel.script

    MEMORY {
      /* Previously: QEMU boots at 0x4028_0000 */
      dram (rwx)  : ORIGIN = 0x40800000, LENGTH = 2M
    
      /* Previously: QEMU Paged Memory is at 0x4028_0000 */
      pgram (rwx) : ORIGIN = 0x40A00000, LENGTH = 4M   /* w/ cache */
    
      /* Why? Because 0x4080_0000 + 2 MB = 0x40A0_0000 */

    (Note that Paged Memory Pool shifts down)

  2. Since we changed the Paged Memory Pool (pgram), we update ARCH_PGPOOL_PBASE and VBASE: configs/knsh/defconfig

    ## Physical Address of Paged Memory Pool
    ## Previously: QEMU Paged Memory is at 0x4028_0000
    CONFIG_ARCH_PGPOOL_PBASE=0x40A00000
    
    ## Virtual Address of Paged Memory Pool
    ## Previously: QEMU Paged Memory is at 0x4028_0000
    CONFIG_ARCH_PGPOOL_VBASE=0x40A00000

    (Paged Memory Pool shall be dished out as Virtual Memory to NuttX Apps)

  3. NuttX QEMU declares the RAM Size as 128 MB in RAMBANK1_SIZE. We set RAM_SIZE accordingly: configs/knsh/defconfig

    ## RAM Size is a paltry 128 MB
    CONFIG_RAM_SIZE=134217728

    (Kinda tiny, but sufficient)

  4. Linux Kernel Header has an incorrect Image Load Offset. Arm64 Bootloaders don’t care, we’ll let it be…

    /* Bootloader starts NuttX here, followed by Linux Kernel Header */
    __start:
      ...
      /* Image Load Offset from Start of RAM         */
      /* Boot Address - CONFIG_RAM_START = 0x800000  */
      /* But we won't change this, since it's unused */
      .quad 0x800000

With these mods, our C Code in NuttX shall boot correctly. FYI: Boot Address also appears on the Onboard LCD…

Avaota-A1 SBC with Onboard LCD

§6 UART Driver for 16550

Our C Code can print to UART now?

To watch the Boot Progress (Sesame Street-style), we can print primitively to UART like this: qemu_boot.c

// 0x0250_0000 is the UART0 Base Address
void arm64_boot_primary_c_routine(void) {
  *(volatile uint8_t *) 0x02500000 = 'A';
  arm64_chip_boot();
  ...

void arm64_chip_boot(void) {
  *(volatile uint8_t *) 0x02500000 = 'B';
  arm64_mmu_init(true);  // Init the Memory Mgmt Unit

  *(volatile uint8_t *) 0x02500000 = 'C';
  arm64_enable_mte();    // Init the Memory Tagging Extension

  *(volatile uint8_t *) 0x02500000 = 'D';
  qemu_board_initialize();  // Init the Board

  *(volatile uint8_t *) 0x02500000 = 'E';
  arm64_earlyserialinit();  // Init the Serial Driver

  *(volatile uint8_t *) 0x02500000 = 'F';
  syslog_rpmsg_init_early(...);  // Init the System Logger

  *(volatile uint8_t *) 0x02500000 = 'G';
  up_perf_init(..);  // Init the Performance Counters

Beyond Big Bird: We need the 16550 UART Driver. Based on the A527 UART Doc

  1. NuttX Boot Code (Arm64 Assembly) will print to UART. We patch it: qemu_lowputc.S

    // Base Address and Baud Rate for 16550 UART
    #define UART1_BASE_ADDRESS          0x02500000
    #define EARLY_UART_PL011_BAUD_RATE  115200
  2. NuttX Boot Code will drop UART Output, unless we wait for UART Ready: qemu_lowputc.S

    /* Wait for 16550 UART to be ready to transmit
    * xb: Register that contains the UART Base Address
    * wt: Scratch register number */
    .macro early_uart_ready xb, wt
    1:
      ldrh  \wt, [\xb, #0x14] /* UART_LSR (Line Status Register) */
      tst   \wt, #0x20        /* Check THRE (TX Holding Register Empty) */
      b.eq  1b                /* Wait for the UART to be ready (THRE=1) */
    .endm

    (Thanks to PinePhone)

  3. QEMU uses PL011 UART. We switch to 16550 UART: qemu_serial.c

    // Switch from PL011 UART (QEMU) to 16550 UART
    #include <nuttx/serial/uart_16550.h>
    
    // Enable the 16550 Console UART at Startup
    void arm64_earlyserialinit(void) {
      // Previously for QEMU: pl011_earlyserialinit
      u16550_earlyserialinit();
    }
    
    // Ditto but not so early
    void arm64_serialinit(void) {
      // Previous for QEMU: pl011_serialinit
      u16550_serialinit();
    }
  4. 16550 UART shall be configured: configs/knsh/defconfig

    CONFIG_16550_ADDRWIDTH=0
    CONFIG_16550_REGINCR=4
    CONFIG_16550_UART0=y
    CONFIG_16550_UART0_BASE=0x02500000
    CONFIG_16550_UART0_CLOCK=198144000
    CONFIG_16550_UART0_IRQ=125
    CONFIG_16550_UART0_SERIAL_CONSOLE=y
    CONFIG_16550_UART=y
    CONFIG_16550_WAIT_LCR=y
    CONFIG_SERIAL_UART_ARCH_MMIO=y
  5. PL011 UART shall be removed: configs/knsh/defconfig

    ## Remove PL011 UART from NuttX Config:
    ## CONFIG_UART1_BASE=0x9000000
    ## CONFIG_UART1_IRQ=33
    ## CONFIG_UART1_PL011=y
    ## CONFIG_UART1_SERIAL_CONSOLE=y
    ## CONFIG_UART_PL011=y
  6. 16550_UART0_CLOCK isn’t quite correct, we’ll settle later. Meanwhile we disable the UART Clock Configuration: uart_16550.c

    // We disable the UART Clock Configuration...
    static int u16550_setup(FAR struct uart_dev_s *dev) { ...
    #ifdef FIX_LATER  // We'll fix it later
      // Enter DLAB=1
      u16550_serialout(priv, UART_LCR_OFFSET, (lcr | UART_LCR_DLAB));
    
      // Omitted: Set the UART Baud Divisor
      // ...
    
      // Clear DLAB
      u16550_serialout(priv, UART_LCR_OFFSET, lcr);
    #endif

Same old drill: Rebuild, recopy and reboot NuttX. We see plenty more debug output

123
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
AB

OK the repeated rebuilding, recopying and rebooting of NuttX is getting really tiresome. We automate…

Avaota-A1 SBC with SDWire MicroSD Multiplexer and Smart Power Plug

§7 MicroSD Multiplexer + Smart Power Plug

What if we could rebuild-recopy-reboot NuttX… In One Single Script?

Thankfully our Avaota-A1 SBC is connected to SDWire MicroSD Multiplexer and Smart Power Plug (pic above). Our Build Script shall do everything for us…

  1. Copy NuttX to MicroSD

  2. Swap the MicroSD from our Test PC to SBC

  3. Power up SBC and boot NuttX

  4. Run OSTest and verify the result

  5. How it looks? Watch the Demo

Avaota-A1 SBC with SDWire MicroSD Multiplexer and Smart Power Plug

All this happens in our nifty Build Script: run.sh

## Build NuttX and Apps (NuttX Kernel Build)
make -j
make -j export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make -j import
popd

## Generate the Initial RAM Disk
## Prepare a Padding with 64 KB of zeroes
## Append Padding and Initial RAM Disk to the NuttX Kernel
genromfs -f initrd -d ../apps/bin -V "NuttXBootVol"
head -c 65536 /dev/zero >/tmp/nuttx.pad
cat nuttx.bin /tmp/nuttx.pad initrd \
  >Image

## Get the Home Assistant Token
## That we copied from http://localhost:8123/profile/security
## export token=xxxx
. $HOME/home-assistant-token.sh

## Power Off the SBC
curl \
  -X POST \
  -H "Authorization: Bearer $token" \
  -H "Content-Type: application/json" \
  -d '{"entity_id": "automation.avaota_power_off"}' \
  http://localhost:8123/api/services/automation/trigger

## Copy NuttX Image to MicroSD
## No password needed for sudo, see below
## Change `thinkcentre` to your Test PC
## https://github.com/lupyuen/nuttx-avaota-a1/blob/main/copy-image.sh
scp Image thinkcentre:/tmp/Image
ssh thinkcentre ls -l /tmp/Image
ssh thinkcentre sudo /home/user/copy-image.sh

## Power On the SBC
curl \
  -X POST \
  -H "Authorization: Bearer $token" \
  -H "Content-Type: application/json" \
  -d '{"entity_id": "automation.avaota_power_on"}' \
  http://localhost:8123/api/services/automation/trigger

## Wait for SBC to finish booting
sleep 30

## Power Off the SBC
curl \
  -X POST \
  -H "Authorization: Bearer $token" \
  -H "Content-Type: application/json" \
  -d '{"entity_id": "automation.avaota_power_off"}' \
  http://localhost:8123/api/services/automation/trigger

(See the Build Log)

(Watch the Demo on YouTube)

Smart Power Plug in IKEA App and Google Home

This script assumes that we have…

Smart Power Plug in Home Assistant

What’s copy_image.sh?

This is the script that copies our NuttX Image to MicroSD, via the SDWire MicroSD Multiplexer, explained here…

Can we automate the testing? Including OSTest?

Yep we have an Expect Script that will execute and verify OSTest: run.sh

## Copy NuttX Image to MicroSD
## No password needed for sudo, see below
## Change `thinkcentre` to your Test PC
## https://github.com/lupyuen/nuttx-avaota-a1/blob/main/copy-image.sh
scp Image thinkcentre:/tmp/Image
ssh thinkcentre ls -l /tmp/Image
ssh thinkcentre sudo /home/user/copy-image.sh

## Boot and Test NuttX with OSTest
## https://github.com/lupyuen/nuttx-build-farm/blob/main/avaota.exp
export AVAOTA_SERVER=thinkcentre
pushd $HOME/nuttx-build-farm
expect ./avaota.exp
popd

§8 Arm64 Memory Management Unit

It’s getting late. Can we get back to NuttX now?

24 Hours is all we need no worries! Earlier we saw NuttX stuck at “AB”

123
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
AB

Which says that NuttX is stranded inside arm64_mmu_init: qemu_boot.c

// 0x0250_0000 is the UART0 Base Address
void arm64_boot_primary_c_routine(void) {
  *(volatile uint8_t *) 0x02500000 = 'A';
  arm64_chip_boot();
  ...

// `AB` means that NuttX is stuck inside arm64_mmu_init()
void arm64_chip_boot(void) {
  *(volatile uint8_t *) 0x02500000 = 'B';
  arm64_mmu_init(true);  // Init the Memory Mgmt Unit

  // Stuck above, never came here
  *(volatile uint8_t *) 0x02500000 = 'C';
  arm64_enable_mte();    // Init the Memory Tagging Extension

What’s arm64_mmu_init?

NuttX calls arm64_mmu_init to start the Arm64 Memory Management Unit (MMU). We add some logs inside: arm64_mmu.c

// Enable Debugging for MMU
#define CONFIG_MMU_ASSERT 1
#define CONFIG_MMU_DEBUG  1
#define trace_printf _info

// We fix the Debug Output, changing `%lux` to `%p`
static void init_xlat_tables(const struct arm_mmu_region *region) {
  ...
  sinfo("mmap: virt %p phys %p size %p\n", virt, phys, size);

// To enable the MMU at Exception Level 1...
static void enable_mmu_el1(unsigned int flags) {
  ...
  // Flush the Cached Data before Enabling MMU
  _info("UP_MB");
  UP_MB();

  // Enable the MMU and Data Cache
  _info("Enable the MMU and data cache");
  write_sysreg(value | SCTLR_M_BIT | SCTLR_C_BIT, sctlr_el1);

  // Ensure that MMU Enable takes effect immediately
  _info("UP_ISB");
  UP_ISB();

And we enable the logs for Scheduler and Memory Manager: configs/knsh/defconfig

## Enable Logging for Memory Manager
CONFIG_DEBUG_MM=y
CONFIG_DEBUG_MM_ERROR=y
CONFIG_DEBUG_MM_INFO=y
CONFIG_DEBUG_MM_WARN=y

## Enable Logging for Scheduler
CONFIG_DEBUG_SCHED=y
CONFIG_DEBUG_SCHED_ERROR=y
CONFIG_DEBUG_SCHED_INFO=y
CONFIG_DEBUG_SCHED_WARN=y

Ah OK we’re stuck just before Enabling the MMU

## Init the MMU Page Translation Tables
init_xlat_tables: mmap: virt 0x7000000    phys 0x7000000    size 0x20000000
init_xlat_tables: mmap: virt 0x40000000   phys 0x40000000   size 0x8000000
init_xlat_tables: mmap: virt 0x4010000000 phys 0x4010000000 size 0x10000000
init_xlat_tables: mmap: virt 0x8000000000 phys 0x8000000000 size 0x8000000000
init_xlat_tables: mmap: virt 0x3eff0000 phys 0x3eff0000 size 0x10000
init_xlat_tables: mmap: virt 0x40800000 phys 0x40800000 size 0x2a000
init_xlat_tables: mmap: virt 0x4082a000 phys 0x4082a000 size 0x6000
init_xlat_tables: mmap: virt 0x40830000 phys 0x40830000 size 0x13000
init_xlat_tables: mmap: virt 0x40a00000 phys 0x40a00000 size 0x400000

## Enable the MMU at Exception Level 1
enable_mmu_el1: UP_MB
enable_mmu_el1: Enable the MMU and data cache

(Exception Level explained)

Something sus about the above Mystery Addresses, what are they?

VirtualPhysicalSize
0x0700_00000x0700_00000x2000_0000
0x4000_00000x4000_00000x0800_0000
0x40_1000_00000x40_1000_00000x1000_0000
0x80_0000_00000x80_0000_00000x80_0000_0000
0x3EFF_00000x3EFF_00000x0001_0000
0x4080_00000x4080_00000x0002_A000
0x4082_A0000x4082_A0000x0000_6000
0x4083_00000x4083_00000x0001_3000
0x40A0_00000x40A0_00000x0040_0000

A527 Memory Map

§9 Fix the Memory Map

Why do we need Arm64 MMU? (Memory Management Unit)

We require MMU for…

If we don’t configure MMU with the correct Memory Map

Arm64 MMU won’t turn on. Maybe our Memory Map is incorrect?

We verify our A527 Memory Map (pic above)

A523 User ManualPage 40
ModuleAddress
Boot ROM & SRAM0x0000_0000 to …
PCIE0x2000_0000 to 0x2FFF_FFFF
DRAM0x4000_0000 to …

How does this compare with NuttX? We do extra MMU Logging: arm64_mmu.c

// Print the Names of the MMU Memory Regions
static void init_xlat_tables(const struct arm_mmu_region *region) { ...
  _info("name=%s\n", region->name);
  sinfo("mmap: virt %p phys %p size %p\n", virt, phys, size);

Ah much clearer! Now we see the Names of Memory Regions for the MMU…

NamePhysicalSize
DEVICE_REGION0x0700_00000x2000_0000
DRAM0_S00x4000_00000x0800_0000
PCI_CFG0x40_1000_00000x1000_0000
PCI_MEM0x80_0000_00000x80_0000_0000
PCI_IO0x3EFF_00000x0001_0000
nx_code0x4080_00000x0002_A000
nx_rodata0x4082_A0000x0000_6000
nx_data0x4083_00000x0001_3000
nx_pgpool0x40A0_00000x0040_0000

Two Tweaks…

The rest are hunky dory…

We rebuild, recopy, reboot NuttX. Our Memory Map looks much better now

NamePhysicalSize
DEVICE_REGION0x0000_00000x4000_0000
DRAM0_S00x4000_00000x0800_0000
nx_code0x4080_00000x0002_A000
nx_rodata0x4082_A0000x0000_6000
nx_data0x4083_00000x0001_3000
nx_pgpool0x40A0_00000x0040_0000

Though it crashes elsewhere…

§10 Arm64 Generic Interrupt Controller

Why is NuttX failing with an Undefined Instruction?

gic_validate_dist_version: No GIC version detect
arm64_gic_initialize: no distributor detected, giving up ret=-19
...
nx_start_application: Starting init task: /system/bin/init
arm64_el1_undef: Undefined instruction at 0x408276e4, dump:
Assertion failed panic: at file: common/arm64_fatal.c:572

Yeah this failure is totally misleading. Real Culprit: NuttX couldn’t Init the GIC

No GIC version detect
No distributor detected, giving up

What’s this GIC?

It’s the Arm64 Generic Interrupt Controller (GIC), version 3. GIC shall…

GIC is here…

A523 User ManualPage 263
ModuleBase Address
GIC0x0340_0000

Which has these GIC Registers inside, handling 8 Arm64 Cores…

A523 User ManualPage 264
OffsetRegister
0x00_0000GICD_CTLR (Distributor Control Register)
0x06_0000GICR_CTLR_C0 (Redistributor Control Register, Core 0)
0x08_0000GICR_CTLR_C1 (Ditto, Core 1)
0x0A_0000GICR_CTLR_C2 (Ditto, Core 2)
0x0C_0000GICR_CTLR_C3 (Ditto, Core 3)
0x0E_0000GICR_CTLR_C4 (Ditto, Core 4)
0x10_0000GICR_CTLR_C5 (Ditto, Core 5)
0x12_0000GICR_CTLR_C6 (Ditto, Core 6)
0x14_0000GICR_CTLR_C7 (Ditto, Core 7)
0x16_0000GICDA_CTLR (Distributor Control Register A)

Based on the above, we set the Addresses of GICD and GICR (Distributor / Redistributor): qemu/chip.h

// Base Address of GIC Distributor and Redistributor
#define CONFIG_GICD_BASE   0x3400000
#define CONFIG_GICR_BASE   0x3460000

// Spaced 0x20000 bytes per Arm64 Core
#define CONFIG_GICR_OFFSET 0x20000

Remember to Disable Memory Manager Logging. NuttX GIC Driver starts correctly and complains no more!

## SPI = Physical Interrupt Signal (not the typical SPI)
gic_validate_dist_version:
  GICv3 version detect
  GICD_TYPER = 0x7b0408
  256 SPIs implemented

We’ll call GIC to handle UART Interrupts. Before that: We need NSH Shell…

NuttX Apps Filesystem on ROMFS

§11 NuttX Apps Filesystem

Are we done yet?

For a Simple NuttX Port (Flat Build): Congrats, just fix the UART Interrupt and we’re done!

However we’re doing NuttX Kernel Build. Which needs more work

nx_start_application:
  Starting init task: /system/bin/init
arm64_el1_undef:
  Undefined instruction at 0x408274a4
Assertion failed panic:
  common/arm64_fatal.c:572

What’s /system/bin/init? Why is it failing?

/system/bin/init is NSH Shell. NuttX Kernel Build will load NuttX Apps from a Local Filesystem, and it’s missing right now. (NuttX Flat Build will bind binary Apps directly into Kernel)

To solve this: We bundle the NuttX Apps together into a ROMFS Filesystem

## Generate the Initial RAM Disk
genromfs \
  -f initrd \
  -d ../apps/bin \
  -V "NuttXBootVol"

Then we package NuttX Kernel + NuttX Apps into a NuttX Image

## Prepare a Padding with 64 KB of zeroes
## Append Padding and Initial RAM Disk to the NuttX Kernel
head -c 65536 /dev/zero >/tmp/nuttx.pad
cat nuttx.bin /tmp/nuttx.pad initrd \
  >Image

(See the Build Script)

When NuttX Boots: It will…

  1. Locate the ROMFS Filesystem in memory

  2. Copy it to the designated Memory Region

  3. Mount it as a RAM Disk

  4. Allowing NuttX Kernel to start NSH Shell

    (And other NuttX Apps)

How? See this…

NSH Prompt still missing? It won’t appear until we handle the UART Interrupt…

NuttX on Avaota-A1

§12 Fix the UART Interrupt

One Last Thing: Settle the UART Interrupt and we’re done!

A523 User ManualPage 256
Interrupt NumberInterrupt Source
34UART0

This is how we set the UART0 Interrupt and watch for keypresses: configs/knsh/defconfig

## Set the UART0 Interrupt to 34
CONFIG_16550_UART0_IRQ=34

To Wrap Up: We Disable Logging for Memory Manager and Scheduler. And Disable MMU Debugging.

NSH Prompt finally appears and OSTest completes successfully. Our NuttX Porting is complete yay!

NuttShell (NSH) NuttX-12.4.0
nsh> uname -a
NuttX 12.4.0 6c5c1a5f9f-dirty Mar  8 2025 21:57:02 arm64 qemu-armv8a

nsh> ostest
...
user_main: Exiting
ostest_main: Exiting with status 0

(See the Complete Log)

(See the Final Code)

(Ready for NuttX Upstreaming)

NSH Prompt won’t appear if UART Interrupt is disabled?

That’s because NSH runs as a NuttX App in User Space. When NSH Shell prints this…

NuttShell (NSH) NuttX-12.4.0
nsh>

It calls the Serial Driver. Which will wait for a UART Interrupt to signal that the Transmit Buffer is empty and available.

Thus if UART Interrupt is disabled, nothing gets printed in NuttX Apps. (Explained here)

NuttX might run OK on Radxa Cubie A5E (Allwinner T527)

NuttX might run OK on Radxa Cubie A5E (Allwinner T527)

§13 What’s Next

Right now we’re upstreaming Avatoa-A1 SBC to NuttX Mainline

We’re seeking volunteers to build NuttX Drivers for Avaota-A1 (GPIO, SPI, I2C, MIPI CSI / DSI, Ethernet, WiFi, …) Please lemme know, maybe we can draw something on the Onboard LCD!

Special Thanks to My Sponsors for supporting my writing. Your support means so much to me 🙏

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.org/src/avaota.md

Yuzuki Avaota-A1 SBC with PinePhone MicroSD Extender

§14 Appendix: Build NuttX for Avaota-A1

To boot NuttX on Avatoa-A1: We may download Image from here…

Or follow these steps to compile our (Work-In-Progress) NuttX for Avaota-A1: run.sh

## Download Source Code for NuttX and Apps
git clone https://github.com/lupyuen2/wip-nuttx nuttx --branch avaota2
git clone https://github.com/lupyuen2/wip-nuttx-apps apps --branch avaota2
cd nuttx

## Build NuttX and Apps (NuttX Kernel Build)
tools/configure.sh avaota-a1:nsh
make -j
make -j export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make -j import
popd

## Generate the Initial RAM Disk
## Prepare a Padding with 64 KB of zeroes
## Append Padding and Initial RAM Disk to the NuttX Kernel
genromfs -f initrd -d ../apps/bin -V "NuttXBootVol"
head -c 65536 /dev/zero >/tmp/nuttx.pad
cat nuttx.bin /tmp/nuttx.pad initrd \
  >Image

Read on to boot the NuttX Image on our SBC…

(See the Build Log)

NuttX on Avaota-A1

§15 Appendix: Boot NuttX on Avaota-A1

Earlier we built NuttX for Avaota-A1 and created the Image file, containing the NuttX Kernel + NuttX Apps. Let’s boot it on MicroSD…

  1. Prepare the AvaotaOS MicroSD

    “Boot NuttX on our SBC”

  2. Copy the NuttX Image to MicroSD…

    ## Copy NuttX Image to AvaotaOS MicroSD
    ## Overwrite the `Image` file
    mv \
      /media/$USER/boot/Image \
      /media/$USER/boot/Image.old
    cp \
      Image \
      /media/$USER/boot/Image
    
    ## Unmount and boot it on Avaota-A1
    ls -l /media/$USER/boot/Image
    umount /media/$USER/boot
  3. Boot the MicroSD on our SBC

We can automate the last two steps with a MicroSD Multiplexer and Smart Power Plug: run.sh

## Get the Home Assistant Token
## That we copied from http://localhost:8123/profile/security
## export token=xxxx
. $HOME/home-assistant-token.sh

## Power Off the SBC
curl \
  -X POST \
  -H "Authorization: Bearer $token" \
  -H "Content-Type: application/json" \
  -d '{"entity_id": "automation.avaota_power_off"}' \
  http://localhost:8123/api/services/automation/trigger

## Copy NuttX Image to MicroSD
## No password needed for sudo, see below
## Change `thinkcentre` to your Test PC
## https://github.com/lupyuen/nuttx-avaota-a1/blob/main/copy-image.sh
scp Image thinkcentre:/tmp/Image
ssh thinkcentre ls -l /tmp/Image
ssh thinkcentre sudo /home/user/copy-image.sh

## Power On the SBC
curl \
  -X POST \
  -H "Authorization: Bearer $token" \
  -H "Content-Type: application/json" \
  -d '{"entity_id": "automation.avaota_power_on"}' \
  http://localhost:8123/api/services/automation/trigger

## Wait for SBC to finish testing
echo Press Enter to Power Off
read

## Power Off the SBC
curl \
  -X POST \
  -H "Authorization: Bearer $token" \
  -H "Content-Type: application/json" \
  -d '{"entity_id": "automation.avaota_power_off"}' \
  http://localhost:8123/api/services/automation/trigger

Or to Automate Everything including OSTest: run.sh

## Copy NuttX Image to MicroSD
## No password needed for sudo, see below
## Change `thinkcentre` to your Test PC
## https://github.com/lupyuen/nuttx-avaota-a1/blob/main/copy-image.sh
scp Image thinkcentre:/tmp/Image
ssh thinkcentre ls -l /tmp/Image
ssh thinkcentre sudo /home/user/copy-image.sh

## Boot and Test NuttX with OSTest
## https://github.com/lupyuen/nuttx-build-farm/blob/main/avaota.exp
export AVAOTA_SERVER=thinkcentre
pushd $HOME/nuttx-build-farm
expect ./avaota.exp
popd

(Watch the Demo on YouTube)

(copy-image.sh is explained here)

(Smart Power Plug also)

NuttX boots to NSH Shell. And passes OSTest yay!

NOTICE:  BL31: v2.5(debug):9241004a9
NOTICE:  BL31: Built : 13:37:46, Nov 16 2023
NOTICE:  BL31: No DTB found.
NOTICE:  [SCP] :wait arisc ready....
NOTICE:  [SCP] :arisc version: []
NOTICE:  [SCP] :arisc startup ready
NOTICE:  [SCP] :arisc startup notify message feedback
NOTICE:  [SCP] :sunxi-arisc driver is starting
ERROR:   Error initializing runtime service opteed_fast
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

NuttShell (NSH) NuttX-12.4.0
nsh> uname -a
NuttX 12.4.0 42c0ed4a89 Mar 13 2025 09:13:56 arm64 avaota-a1

nsh> ls -l /dev
/dev:
 crw-rw-rw-           0 console
 crw-rw-rw-           0 null
 brw-rw-rw-    16777216 ram0
 crw-rw-rw-           0 ttyS0
 crw-rw-rw-           0 zero

nsh> ps
  PID GROUP PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK            STACK    USED FILLED COMMAND
    0     0   0 FIFO     Kthread   - Ready              0000000000000000 0008176 0000928  11.3%  Idle_Task
    1     0 192 RR       Kthread   - Waiting  Semaphore 0000000000000000 0008112 0000992  12.2%  hpwork 0x40833568 0x408335b8
    2     0 100 RR       Kthread   - Waiting  Semaphore 0000000000000000 0008112 0000992  12.2%  lpwork 0x408334e8 0x40833538
    4     4 100 RR       Task      - Running            0000000000000000 0008128 0002192  26.9%  /system/bin/init

nsh> free
      total       used       free    maxused    maxfree  nused  nfree name
  125542400      33880  125508520      53032  125484976     58      5 Kmem
    4194304     245760    3948544               3948544               Page

nsh> hello
Hello, World!!

nsh> getprime
Set thread priority to 10
Set thread policy to SCHED_RR
Start thread #0
thread #0 started, looking for primes < 10000, doing 10 run(s)
thread #0 finished, found 1230 primes, last one was 9973
Done
getprime took 162 msec

nsh> ostest
...
Final memory usage:
VARIABLE  BEFORE   AFTER
======== ======== ========
arena        a000    26000
ordblks         2        4
mxordblk     6ff8    1aff8
uordblks     27e8     6700
fordblks     7818    1f900
user_main: Exiting
ostest_main: Exiting with status 0
nsh>

(See the NuttX Log)

Upstreaming NuttX for Avaota-A1

§16 Appendix: Upstream NuttX for Avaota-A1

In this article we ported NuttX QEMU Arm64 (Kernel Build) iteratively to Avaota-A1. Up Next: Upstreaming our code to NuttX Mainline!

Here’s how we copy-n-pasted our Modified Files into a proper NuttX Arch (Allwinner A527) and NuttX Board (Avaota-A1)

  1. Copy qemu folders to a527. Copy qemu-armv8a folder to avaota-a1.

  2. Rename qemu files to a527. Rename qemu-armv8a files to avaota-a1.

  3. Rewrite “qemu-armv8a” to “avaota-a1”. Rewrite “qemu” to “a527”.

  4. Rewrite “A527_ARMV8A” to “AVAOTA_A1”

  5. Add the Arch and Board

  6. Apply the changes from github.com/lupyuen2/wip-nuttx/pull/98/files
    OSTest passes yay!

  7. Remove the unused NuttX Configs

  8. Rename knsh to nsh

  9. Add the Arch and Board Docs

  10. And Much More

Upstreaming becomes lotsa copypasta…

  1. We create a Staging PR in our own repo…

    github.com/lupyuen2/wip-nuttx/pull/99

  2. Dump the list of Modified Files

    ## Change this to our Staging PR
    $ pr=https://github.com/lupyuen2/wip-pinephone-nuttx/pull/99
    $ curl -L $pr.diff \
      | grep "diff --git" \
      | sort \
      | cut -d" " -f3 \
      | cut -c3-
    
    ## Here are the Modified Files for our PR
    Documentation/platforms/arm64/a527/boards/avaota-a1/avaota-a1.jpg
    Documentation/platforms/arm64/a527/boards/avaota-a1/index.rst
    Documentation/platforms/arm64/a527/index.rst
    arch/arm64/Kconfig
    arch/arm64/include/a527/chip.h
    arch/arm64/include/a527/irq.h
    arch/arm64/src/a527/CMakeLists.txt
    arch/arm64/src/a527/Kconfig
    arch/arm64/src/a527/Make.defs
    arch/arm64/src/a527/a527_boot.c
    arch/arm64/src/a527/a527_boot.h
    arch/arm64/src/a527/a527_initialize.c
    arch/arm64/src/a527/a527_lowputc.S
    arch/arm64/src/a527/a527_serial.c
    arch/arm64/src/a527/a527_textheap.c
    arch/arm64/src/a527/a527_timer.c
    arch/arm64/src/a527/chip.h
    boards/Kconfig
    boards/arm64/a527/avaota-a1/CMakeLists.txt
    boards/arm64/a527/avaota-a1/Kconfig
    boards/arm64/a527/avaota-a1/configs/nsh/defconfig
    boards/arm64/a527/avaota-a1/include/board.h
    boards/arm64/a527/avaota-a1/include/board_memorymap.h
    boards/arm64/a527/avaota-a1/scripts/Make.defs
    boards/arm64/a527/avaota-a1/scripts/gnu-elf.ld
    boards/arm64/a527/avaota-a1/scripts/ld.script
    boards/arm64/a527/avaota-a1/src/CMakeLists.txt
    boards/arm64/a527/avaota-a1/src/Makefile
    boards/arm64/a527/avaota-a1/src/a527_appinit.c
    boards/arm64/a527/avaota-a1/src/a527_boardinit.c
    boards/arm64/a527/avaota-a1/src/a527_bringup.c
    boards/arm64/a527/avaota-a1/src/a527_power.c
    boards/arm64/a527/avaota-a1/src/avaota-a1.h
  3. Check nxstyle on the Modified Files…

    ## Run nxstyle on the Modified Files
    nxstyle Documentation/platforms/arm64/a527/boards/avaota-a1/avaota-a1.jpg
    nxstyle Documentation/platforms/arm64/a527/boards/avaota-a1/index.rst
    nxstyle Documentation/platforms/arm64/a527/index.rst
    nxstyle arch/arm64/Kconfig
    nxstyle arch/arm64/include/a527/chip.h
    nxstyle arch/arm64/include/a527/irq.h
    nxstyle arch/arm64/src/a527/CMakeLists.txt
    nxstyle arch/arm64/src/a527/Kconfig
    nxstyle arch/arm64/src/a527/Make.defs
    nxstyle arch/arm64/src/a527/a527_boot.c
    nxstyle arch/arm64/src/a527/a527_boot.h
    nxstyle arch/arm64/src/a527/a527_initialize.c
    nxstyle arch/arm64/src/a527/a527_lowputc.S
    nxstyle arch/arm64/src/a527/a527_serial.c
    nxstyle arch/arm64/src/a527/a527_textheap.c
    nxstyle arch/arm64/src/a527/a527_timer.c
    nxstyle arch/arm64/src/a527/chip.h
    nxstyle boards/Kconfig
    nxstyle boards/arm64/a527/avaota-a1/CMakeLists.txt
    nxstyle boards/arm64/a527/avaota-a1/Kconfig
    nxstyle boards/arm64/a527/avaota-a1/configs/nsh/defconfig
    nxstyle boards/arm64/a527/avaota-a1/include/board.h
    nxstyle boards/arm64/a527/avaota-a1/include/board_memorymap.h
    nxstyle boards/arm64/a527/avaota-a1/scripts/Make.defs
    nxstyle boards/arm64/a527/avaota-a1/scripts/gnu-elf.ld
    nxstyle boards/arm64/a527/avaota-a1/scripts/ld.script
    nxstyle boards/arm64/a527/avaota-a1/src/CMakeLists.txt
    nxstyle boards/arm64/a527/avaota-a1/src/Makefile
    nxstyle boards/arm64/a527/avaota-a1/src/a527_appinit.c
    nxstyle boards/arm64/a527/avaota-a1/src/a527_boardinit.c
    nxstyle boards/arm64/a527/avaota-a1/src/a527_bringup.c
    nxstyle boards/arm64/a527/avaota-a1/src/a527_power.c
    nxstyle boards/arm64/a527/avaota-a1/src/avaota-a1.h
  4. Copy the Arch Files into the Arch Pull Request

    “arch/arm64/a527: Add support for Allwinner A527 SoC”

    ## Download the Branch for Avaota Arch (initially empty)
    pushd /tmp 
    git clone https://github.com/lupyuen2/wip-nuttx avaota-arch --branch avaota-arch
    popd
    
    ## Copy the Arch Files from src to dest
    function copy_files() {
      src=.
      dest=/tmp/avaota-arch
      for file in \
        Documentation/platforms/arm64/a527/index.rst \
        arch/arm64/Kconfig \
        arch/arm64/include/a527/chip.h \
        arch/arm64/include/a527/irq.h \
        arch/arm64/src/a527/CMakeLists.txt \
        arch/arm64/src/a527/Kconfig \
        arch/arm64/src/a527/Make.defs \
        arch/arm64/src/a527/a527_boot.c \
        arch/arm64/src/a527/a527_boot.h \
        arch/arm64/src/a527/a527_initialize.c \
        arch/arm64/src/a527/a527_lowputc.S \
        arch/arm64/src/a527/a527_serial.c \
        arch/arm64/src/a527/a527_textheap.c \
        arch/arm64/src/a527/a527_timer.c \
        arch/arm64/src/a527/chip.h \
    
      do
        src_file=$src/$file
        dest_file=$dest/$file
        dest_dir=$(dirname -- "$dest_file")
        set -x
        mkdir -p $dest_dir
        cp $src_file $dest_file
        set +x
      done
    }
    
    ## Copy and commit /tmp/avaota-arch
    ## Remove the "Supported Boards" (toctree) from Arch Doc
    copy_files
    code /tmp/avaota-arch
  5. Copy the Board Files into the Board Pull Request

    ## Download the Branch for Avaota Board (initially empty)
    pushd /tmp 
    git clone https://github.com/lupyuen2/wip-nuttx avaota-board --branch avaota-board
    popd
    
    ## Copy the Board Files from src to dest
    ## Copy the Arch Doc again because we restored the "Supported Boards" 
    function copy_files() {
      src=.
      dest=/tmp/avaota-board
      for file in \
        Documentation/platforms/arm64/a527/index.rst \
        Documentation/platforms/arm64/a527/boards/avaota-a1/avaota-a1.jpg \
        Documentation/platforms/arm64/a527/boards/avaota-a1/index.rst \
        boards/Kconfig \
        boards/arm64/a527/avaota-a1/CMakeLists.txt \
        boards/arm64/a527/avaota-a1/Kconfig \
        boards/arm64/a527/avaota-a1/configs/nsh/defconfig \
        boards/arm64/a527/avaota-a1/include/board.h \
        boards/arm64/a527/avaota-a1/include/board_memorymap.h \
        boards/arm64/a527/avaota-a1/scripts/Make.defs \
        boards/arm64/a527/avaota-a1/scripts/gnu-elf.ld \
        boards/arm64/a527/avaota-a1/scripts/ld.script \
        boards/arm64/a527/avaota-a1/src/CMakeLists.txt \
        boards/arm64/a527/avaota-a1/src/Makefile \
        boards/arm64/a527/avaota-a1/src/a527_appinit.c \
        boards/arm64/a527/avaota-a1/src/a527_boardinit.c \
        boards/arm64/a527/avaota-a1/src/a527_bringup.c \
        boards/arm64/a527/avaota-a1/src/a527_power.c \
        boards/arm64/a527/avaota-a1/src/avaota-a1.h \
    
      do
        src_file=$src/$file
        dest_file=$dest/$file
        dest_dir=$(dirname -- "$dest_file")
        set -x
        mkdir -p $dest_dir
        cp $src_file $dest_file
        set +x
      done
    }
    
    ## Copy and commit /tmp/avaota-board
    copy_files
    code /tmp/avaota-board
  6. Remember to create Two Commits Per PR: One Commit for Code, Another Commit for Docs

    Two Commits Per PR: One Commit for Code, Another Commit for Docs

  7. Need to Squash the Commits (or amend them), but another Code or Doc Commit is stuck in between?

    Before Reordering the Commit

    Try Reordering the Commits to the top, before squashing or amending.

    After Reordering the Commit

  8. We’re finally ready to Submit our Pull Requests!

    “arch/arm64/a527: Add support for Allwinner A527 SoC”

SDWire MicroSD Multiplexer

§17 Appendix: SDWire MicroSD Multiplexer

Let’s make our Tweak-Build-Test Cycle quicker for NuttX. We use SDWire MicroSD Multiplexer (pic above) to flip our MicroSD between Test PC and SBC

SDWire needs Plenty of Sudo Passwords to flip the multiplexer, mount the filesystem, copy to MicroSD. We make it Sudo Password-Less with visudo

  1. Wrap all the Sudo Commands into a script: copy-image.sh

    ## Create a Bash Script: copy-image.sh
    ## Containing these commands...
    
    set -e  ## Exit when any command fails
    set -x  ## Echo commands
    whoami  ## I am root!
    
    ## Copy /tmp/Image to MicroSD
    sd-mux-ctrl --device-serial=sd-wire_02-09 --ts
    sleep 5
    mkdir -p /tmp/sda1
    mount /dev/sda1 /tmp/sda1
    cp /tmp/Image /tmp/sda1/
    ls -l /tmp/sda1
    
    ## Unmount MicroSD and flip it to the Test Device (Avaota-A1)
    umount /tmp/sda1
    sd-mux-ctrl --device-serial=sd-wire_02-09 --dut
  2. Configure visudo so that our script will run as Sudo Without Password

    ## Make our script executable
    ## Start the Sudoers Editor
    chmod +x /home/user/copy-image.sh
    sudo visudo
    
    ## Add this line:
    user ALL=(ALL) NOPASSWD: /home/user/copy-image.sh
  3. Then we can trigger our script remotely via SSH, Without Sudo Password: run.sh

    ## Copy NuttX Image to MicroSD
    ## No password needed for sudo yay!
    scp Image thinkcentre:/tmp/Image
    ssh thinkcentre \
      ls -l /tmp/Image
    ssh thinkcentre \
      sudo /home/user/copy-image.sh
  4. Everything goes into our Build Script for NuttX

    How it looks? Watch the Demo

(Actually we could allow anyone in the world to Remotely Build and Test NuttX on our Avaota-A1 SBC hmmm…)

NuttX Apps Filesystem in ROMFS

§18 Appendix: NuttX Apps Filesystem

Earlier we talked about the ROMFS Filesystem for NuttX Apps (Initial RAM Disk, pic above)

This section explains how we implemented the NuttX Apps Filesystem…

After this implementation, /system/bin/init (NSH Shell) shall start successfully

qemu_bringup:
mount_ramdisk:
nx_start_application: ret=0
nx_start_application: Starting init task: /system/bin/init
nxtask_activate: /system/bin/init pid=4,TCB=0x408469f0
nxtask_exit: AppBringUp pid=3,TCB=0x40846190
board_app_initialize:
nx_start: CPU0: Beginning Idle Loop

§18.1 HostFS becomes ROMFS

QEMU uses Semihosting and HostFS to access the NuttX Apps Filesystem. We change to ROMFS: configs/knsh/defconfig

## We added ROMFS...
CONFIG_BOARDCTL_ROMDISK=y
CONFIG_BOARD_LATE_INITIALIZE=y
CONFIG_INIT_MOUNT_TARGET="/system/bin"

## And removed Semihosting HostFS...
## CONFIG_FS_HOSTFS=y
## CONFIG_ARM64_SEMIHOSTING_HOSTFS=y
## CONFIG_ARM64_SEMIHOSTING_HOSTFS_CACHE_COHERENCE=y
## CONFIG_INIT_MOUNT_DATA="fs=../apps"
## CONFIG_INIT_MOUNT_FSTYPE="hostfs"
## CONFIG_INIT_MOUNT_SOURCE=""
## CONFIG_INIT_MOUNT_TARGET="/system"

BOARD_LATE_INITIALIZE is needed because we’ll Mount the ROMFS Filesystem inside qemu_bringup. (See below)

§18.2 Linker Script

We reserve 16 MB of RAM for the ROMFS Filesystem that will host the NuttX Apps: ld-kernel.script

/* Linker Script: We added the RAM Disk (16 MB) */
MEMORY {
  dram (rwx)    : ORIGIN = 0x40800000, LENGTH = 2M
  pgram (rwx)   : ORIGIN = 0x40A00000, LENGTH = 4M    /* w/ cache */
  ramdisk (rwx) : ORIGIN = 0x40E00000, LENGTH = 16M   /* w/ cache */
}

/* We'll reference these in our code */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size  = LENGTH(ramdisk);
__ramdisk_end   = ORIGIN(ramdisk) + LENGTH(ramdisk);

§18.3 Mount the ROMFS

At Startup: We mount the ROMFS Filesystem (inside RAM) as /dev/ram0: qemu_bringup.c

// At NuttX Startup...
int qemu_bringup(void) {
  // We Mount the RAM Disk
  mount_ramdisk();
  ...
}

// Mount a RAM Disk defined in ld.script to /dev/ramX.  The RAM Disk
// contains a ROMFS filesystem with applications that can be spawned at
// runtime.
static int mount_ramdisk(void) {
  struct boardioc_romdisk_s desc;
  desc.minor    = RAMDISK_DEVICE_MINOR;
  desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
  desc.sectsize = SECTORSIZE;
  desc.image    = __ramdisk_start;

  int ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);
  if (ret < 0) {
    syslog(LOG_ERR, "Ramdisk register failed: %s\n", strerror(errno));
    syslog(LOG_ERR, "Ramdisk mountpoint /dev/ram%d\n",RAMDISK_DEVICE_MINOR);
    syslog(LOG_ERR, "Ramdisk length %lu, origin %lx\n", (ssize_t)__ramdisk_size, (uintptr_t)__ramdisk_start);
  }
  return ret;
}

// RAM Disk Definition
#define SECTORSIZE   512
#define NSECTORS(b)  (((b) + SECTORSIZE - 1) / SECTORSIZE)
#define RAMDISK_DEVICE_MINOR 0

§18.4 Copy the ROMFS

But Before That: We safely copy the ROMFS Filesystem (Initial RAM Disk) from the NuttX Image into the ramdisk Memory Region

Mounting the ROMFS Filesystem

This happens just after Bootloader starts NuttX: qemu_boot.c

// Needed for the `aligned_data` macro
#include <nuttx/compiler.h>

// Just after Bootloader has started NuttX...
void arm64_chip_boot(void) {

  // We copy the RAM Disk
  qemu_copy_ramdisk();

  // Omitted: Other initialisation (MMU, ...)
  arm64_mmu_init(true);
  ...
}

// Copy the RAM Disk from NuttX Image to RAM Disk Region.
static void qemu_copy_ramdisk(void) {
  const uint8_t aligned_data(8) header[8] = "-rom1fs-";
  const uint8_t *limit = (uint8_t *)g_idle_topstack + (256 * 1024);
  uint8_t *ramdisk_addr = NULL;
  uint8_t *addr;
  uint32_t size;

  // After Idle Stack Top, search for "-rom1fs-". This is the RAM Disk Address.
  // Limit search to 256 KB after Idle Stack Top.
  for (addr = g_idle_topstack; addr < limit; addr += 8) {
      if (memcmp(addr, header, sizeof(header)) == 0) {
        ramdisk_addr = addr;
        break;
      }
  }

  // Stop if RAM Disk is missing
  if (ramdisk_addr == NULL) {
    _err("Missing RAM Disk. Check the initrd padding.");
    PANIC();
  }

  // Read the Filesystem Size from the next 4 bytes (Big Endian)
  size = (ramdisk_addr[8] << 24) + (ramdisk_addr[9] << 16) +
         (ramdisk_addr[10] << 8) + ramdisk_addr[11] + 0x1f0;

  // Filesystem Size must be less than RAM Disk Memory Region
  if (size > (size_t)__ramdisk_size) {
    _err("RAM Disk Region too small. Increase by %lu bytes.\n", size - (size_t)__ramdisk_size);
    PANIC();
  }

  // Copy the RAM Disk from NuttX Image to RAM Disk Region.
  // __ramdisk_start overlaps with ramdisk_addr + size.
  qemu_copy_overlap(__ramdisk_start, ramdisk_addr, size);
}

// Copy an overlapping memory region.  dest overlaps with src + count.
static void qemu_copy_overlap(uint8_t *dest, const uint8_t *src, size_t count) {
  uint8_t *d = dest + count - 1;
  const uint8_t *s = src + count - 1;
  if (dest <= src) { _err("dest and src should overlap"); PANIC(); }
  while (count--) {
    volatile uint8_t c = *s;  // Prevent compiler optimization
    *d = c;
    d--;
    s--;
  }
} 

// RAM Disk Region is defined in Linker Script
extern uint8_t __ramdisk_start[];
extern uint8_t __ramdisk_size[];

(Moved here)

Why the aligned addresses?

// Header is aligned to 8 bytes
const uint8_t
  aligned_data(8) header[8] =
  "-rom1fs-";

// Address is also aligned to 8 bytes
for (
  addr = g_idle_topstack;
  addr < limit;
  addr += 8
) {
  // Otherwise this will hit Alignment Fault
  memcmp(addr, header, sizeof(header));
  ...
}

We align our Memory Accesses to 8 Bytes. Otherwise we’ll hit an Alignment Fault

## Alignment Fault at `memcmp(addr, header, sizeof(header))`
default_fatal_handler:
  (IFSC/DFSC) for Data/Instruction aborts:
  alignment fault

(Strangely: This Alignment isn’t needed for RISC-V)

Porting NuttX to Avaota-A1

§19 Appendix: Port NuttX to Avaota-A1

In this article, we took NuttX for Arm64 QEMU Kernel Build (qemu-armv8a:knsh) and changed it slightly for Avaota-A1 SBC. To help our PR Reviewers: This section explains the Modified Code in our Pull Request…

Only Seven Files were modified from QEMU NuttX. All other files were simply copied and renamed, from QEMU NuttX to Avaota-A1. (Pic above)

§19.1 Memory Map

arch/arm64/include/a527/chip.h

We define the I/O Memory Space

// I/O Memory Space
#define CONFIG_DEVICEIO_BASEADDR   0x00000000
#define CONFIG_DEVICEIO_SIZE       MB(1024)

// Kernel Boot Address from SBC Bootloader
#define CONFIG_LOAD_BASE           0x40800000

Based on the A527 Memory Map

A523 User ManualPage 40
ModuleAddress
Boot ROM & SRAM0x0000_0000 to …
PCIE0x2000_0000 to 0x2FFF_FFFF
DRAM0x4000_0000 to …

(Explained here)

§19.2 GIC Interrupt Controller

arch/arm64/include/a527/chip.h

We set the GIC Base Addresses

// GICD and GICD Base Addresses
#define CONFIG_GICD_BASE           0x3400000
#define CONFIG_GICR_BASE           0x3460000

Based on the GIC Doc

A523 User ManualPage 263
ModuleBase Address
GIC0x0340_0000

And GIC Registers

A523 User ManualPage 264
OffsetRegister
0x00_0000GICD_CTLR (Distributor Control Register)
0x06_0000GICR_CTLR_C0 (Redistributor Control Register, Core 0)
0x08_0000GICR_CTLR_C1 (Ditto, Core 1)
0x0A_0000GICR_CTLR_C2 (Ditto, Core 2)
0x0C_0000GICR_CTLR_C3 (Ditto, Core 3)
0x0E_0000GICR_CTLR_C4 (Ditto, Core 4)
0x10_0000GICR_CTLR_C5 (Ditto, Core 5)
0x12_0000GICR_CTLR_C6 (Ditto, Core 6)
0x14_0000GICR_CTLR_C7 (Ditto, Core 7)
0x16_0000GICDA_CTLR (Distributor Control Register A)

(Explained here)


boards/arm64/a527/avaota-a1/configs/nsh/defconfig

We set the UART0 Interrupt

## Set the UART0 Interrupt to 34
CONFIG_16550_UART0_IRQ=34

Based on the A527 Doc…

A523 User ManualPage 256
Interrupt NumberInterrupt Source
34UART0

(Explained here)

§19.3 Arm64 Boot Code

arch/arm64/src/a527/a527_lowputc.S

We updated the Arm64 Boot Code for 16550 UART Driver

  1. We modified the UART Base Address

    // Base Address for 16550 UART
    #define UART0_BASE_ADDRESS 0x02500000
  2. QEMU was using PL011 UART. We fixed this for 16550 UART, to Wait for UART Ready (derived from NuttX A64)

    /* Wait for 16550 UART to be ready to transmit
    * xb: Register that contains the UART Base Address
    * wt: Scratch register number */
    .macro early_uart_ready xb, wt
    1:
      ldrh  \wt, [\xb, #0x14] /* UART_LSR (Line Status Register) */
      tst   \wt, #0x20        /* Check THRE (TX Holding Register Empty) */
      b.eq  1b                /* Wait for the UART to be ready (THRE=1) */
    .endm

UART Base Address came from the A527 Doc…

A523 User ManualPage 1839
ModuleBase Address
UART00x0250_0000

With these UART Registers

A523 User ManualPage 1839
OffsetRegister
0x0000UART_THR (Transmit Holding Register)
0x0004UART_DLH (Divisor Latch High Register)
0x0008UART_IIR (Interrupt Identity Register)
0x000CUART_LCR (Line Control)

(Explained here)

Mounting the ROMFS Filesystem containing the NuttX Apps

§19.4 NuttX Start Code

arch/arm64/src/a527/a527_boot.c

At NuttX Startup: We mount the ROMFS Filesystem (Initial RAM Disk, pic above) containing the NuttX Apps

How? We safely copy the ROMFS Filesystem from the NuttX Image into the ramdisk Memory Region. This code comes from NuttX EIC7700X

// Needed for the `aligned_data` macro
#include <nuttx/compiler.h>

// Just after Bootloader has started NuttX...
void arm64_chip_boot(void) {

  // We copy the RAM Disk
  qemu_copy_ramdisk();

  // Omitted: Other initialisation (MMU, ...)
  arm64_mmu_init(true);
  ...
}

// Copy the RAM Disk from NuttX Image to RAM Disk Region.
static void a527_copy_ramdisk(void) {
  const uint8_t aligned_data(8) header[8] = "-rom1fs-";
  const uint8_t *limit = (uint8_t *)g_idle_topstack + (256 * 1024);
  uint8_t *ramdisk_addr = NULL;
  uint8_t *addr;
  uint32_t size;

  // After Idle Stack Top, search for "-rom1fs-". This is the RAM Disk Address.
  // Limit search to 256 KB after Idle Stack Top.
  for (addr = g_idle_topstack; addr < limit; addr += 8) {
      if (memcmp(addr, header, sizeof(header)) == 0) {
        ramdisk_addr = addr;
        break;
      }
  }

  // Stop if RAM Disk is missing
  if (ramdisk_addr == NULL) {
    _err("Missing RAM Disk. Check the initrd padding.");
    PANIC();
  }

  // Read the Filesystem Size from the next 4 bytes (Big Endian)
  size = (ramdisk_addr[8] << 24) + (ramdisk_addr[9] << 16) +
         (ramdisk_addr[10] << 8) + ramdisk_addr[11] + 0x1f0;

  // Filesystem Size must be less than RAM Disk Memory Region
  if (size > (size_t)__ramdisk_size) {
    _err("RAM Disk Region too small. Increase by %lu bytes.\n", size - (size_t)__ramdisk_size);
    PANIC();
  }

  // Copy the RAM Disk from NuttX Image to RAM Disk Region.
  // __ramdisk_start overlaps with ramdisk_addr + size.
  a527_copy_overlap(__ramdisk_start, ramdisk_addr, size);
}

// Copy an overlapping memory region.  dest overlaps with src + count.
static void a527_copy_overlap(uint8_t *dest, const uint8_t *src, size_t count) {
  uint8_t *d = dest + count - 1;
  const uint8_t *s = src + count - 1;
  if (dest <= src) { _err("dest and src should overlap"); PANIC(); }
  while (count--) {
    volatile uint8_t c = *s;  // Prevent compiler optimization
    *d = c;
    d--;
    s--;
  }
} 

// RAM Disk Region is defined in Linker Script
extern uint8_t __ramdisk_start[];
extern uint8_t __ramdisk_size[];

(Previously here)

Why the aligned addresses?

// Header is aligned to 8 bytes
const uint8_t
  aligned_data(8) header[8] =
  "-rom1fs-";

// Address is also aligned to 8 bytes
for (
  addr = g_idle_topstack;
  addr < limit;
  addr += 8
) {
  // Otherwise this will hit Alignment Fault
  memcmp(addr, header, sizeof(header));
  ...
}

We align our Memory Accesses to 8 Bytes. Otherwise we’ll hit an Alignment Fault

## Alignment Fault at `memcmp(addr, header, sizeof(header))`
default_fatal_handler:
  (IFSC/DFSC) for Data/Instruction aborts:
  alignment fault

(Explained here)

§19.5 Board Bringup Code

boards/arm64/a527/avaota-a1/src/a527_bringup.c

At Board Startup: We mount the ROMFS Filesystem (inside RAM) as /dev/ram0

// At NuttX Startup...
int a527_bringup(void) {
  // We Mount the RAM Disk
  mount_ramdisk();
  ...
}

// Mount a RAM Disk defined in ld.script to /dev/ramX.  The RAM Disk
// contains a ROMFS filesystem with applications that can be spawned at
// runtime.
static int mount_ramdisk(void) {
  struct boardioc_romdisk_s desc;
  desc.minor    = RAMDISK_DEVICE_MINOR;
  desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
  desc.sectsize = SECTORSIZE;
  desc.image    = __ramdisk_start;

  int ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);
  if (ret < 0) {
    syslog(LOG_ERR, "Ramdisk register failed: %s\n", strerror(errno));
    syslog(LOG_ERR, "Ramdisk mountpoint /dev/ram%d\n",RAMDISK_DEVICE_MINOR);
    syslog(LOG_ERR, "Ramdisk length %lu, origin %lx\n", (ssize_t)__ramdisk_size, (uintptr_t)__ramdisk_start);
  }
  return ret;
}

// RAM Disk Definition
#define SECTORSIZE   512
#define NSECTORS(b)  (((b) + SECTORSIZE - 1) / SECTORSIZE)
#define RAMDISK_DEVICE_MINOR 0

(Explained here)

§19.6 Linker Script

boards/arm64/a527/avaota-a1/scripts/ld.script

We reserve 16 MB of RAM for the ROMFS Filesystem that will host the NuttX Apps…

/* Linker Script: We moved the Paged Pool and added the RAM Disk (16 MB) */
MEMORY {
  /* Previously: QEMU boots at 0x4028_0000 */
  dram (rwx)  : ORIGIN = 0x40800000, LENGTH = 2M

  /* Previously: QEMU Paged Memory is at 0x4028_0000 */
  /* Why? Because 0x4080_0000 + 2 MB = 0x40A0_0000   */
  pgram (rwx) : ORIGIN = 0x40A00000, LENGTH = 4M   /* w/ cache */

  /* Added the RAM Disk */
  ramdisk (rwx) : ORIGIN = 0x40E00000, LENGTH = 16M   /* w/ cache */
}

/* We'll reference these in our code */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size  = LENGTH(ramdisk);
__ramdisk_end   = ORIGIN(ramdisk) + LENGTH(ramdisk);

Also we moved the Paged Pool because the Boot Address has changed to 0x4080_0000.

(Explained here)

§19.7 NuttX Config

boards/arm64/a527/avaota-a1/configs/nsh/defconfig

Since we changed the Paged Memory Pool (pgram), we update ARCH_PGPOOL_PBASE and VBASE: configs/knsh/defconfig

## Physical Address of Paged Memory Pool
## Previously: QEMU Paged Memory is at 0x4028_0000
CONFIG_ARCH_PGPOOL_PBASE=0x40A00000

## Virtual Address of Paged Memory Pool
## Previously: QEMU Paged Memory is at 0x4028_0000
CONFIG_ARCH_PGPOOL_VBASE=0x40A00000

(Explained here)

NuttX QEMU declares the RAM Size as 128 MB in RAMBANK1_SIZE. We set RAM_SIZE accordingly…

## RAM Size is 128 MB
CONFIG_RAM_SIZE=134217728

(Explained here)

Based on the 16550 UART Registers above: We configured the 16550 UART and removed PL011 UART…

CONFIG_16550_ADDRWIDTH=0
CONFIG_16550_REGINCR=4
CONFIG_16550_UART0=y
CONFIG_16550_UART0_BASE=0x02500000
CONFIG_16550_UART0_CLOCK=23040000
CONFIG_16550_UART0_IRQ=125
CONFIG_16550_UART0_SERIAL_CONSOLE=y
CONFIG_16550_UART=y
CONFIG_16550_WAIT_LCR=y
CONFIG_SERIAL_UART_ARCH_MMIO=y

(Explained here)

16550_UART0_CLOCK was computed according to these instructions

NuttX UART Debug Log shows:
  dlm = 0x00
  dll = 0x0D

We know that:
  dlm = 0x00 = (div >> 8)
  dll = 0x0D = (div & 0xFF)

Which means:
  div = 0x0D

We know that:
  baud = 115200
  div  = (uartclk + (baud << 3)) / (baud << 4)

Therefore:
  0x0D    = (uartclk + 921600) / 1843200
  uartclk = (0x0D * 1843200) - 921600
          = 23040000

arch/arm64/src/a527/a527_serial.c

QEMU was using PL011 UART. We switched the Serial Driver to 16550 UART

// Switch from PL011 UART (QEMU) to 16550 UART
#include <nuttx/serial/uart_16550.h>

// Enable the 16550 Console UART at Startup
void arm64_earlyserialinit(void) {
  // Previously for QEMU: pl011_earlyserialinit
  u16550_earlyserialinit();
}

// Ditto but not so early
void arm64_serialinit(void) {
  // Previous for QEMU: pl011_serialinit
  u16550_serialinit();
}

(Explained here)