📝 16 Mar 2025
This article explains how we ported NuttX from QEMU Arm64 Kernel Build to PINE64 Yuzuki Avaota-A1 SBC based on Allwinner A527 SoC … Completed within 24 Hours!
Why are we doing this?
Anyone porting NuttX from QEMU to Real SBC? This walkthrough shall be mighty helpful!
Avaota-A1 SBC is Open Source Hardware (CERN OHL Licensed). PINE64 sells it today, maybe we’ll see more manufacturers.
This could be the First Port of Arm64 in NuttX Kernel Build. (NXP i.MX93 might be another?)
We’ll run it as PR Test Bot for validating Arm64 Pull Requests on Real Hardware. PR Test Bot will be fully automated thanks to the MicroSD Multiplexer.
We’re ready for volunteers to build NuttX Drivers for Avaota-A1 / Allwinner A527 (GPIO, SPI, I2C, MIPI CSI / DSI, Ethernet, WiFi, …) Please lemme know, maybe we can draw something on the Onboard LCD! 🙏
Allwinner A523 User Manual (A527 is similar to A523)
(BTW I bought all the hardware covered in this article. Nope, nothing was sponsored: Avaota-A1, SDWire, IKEA TRETAKT)
Nifty Trick for Booting NuttX on Any Arm64 SBC (RISC-V too)
Arm64 Bootloader (U-Boot / SyterKit) will boot Linux by loading the Image
file
(Containing the Linux Kernel)
Thus we “Hijack” the Image
file, replace it by NuttX Kernel
Which means NuttX Kernel shall look and feel like a Linux Kernel
That’s why we have a Linux Kernel Header at the top of NuttX
To begin, we observe our SBC and its Natural Behaviour… How does it Boot Linux?
Connect a USB UART Dongle (CH340 or CP2102) to the UART0 Port (pic above)
Avaota-A1 | USB UART | Colour |
---|---|---|
GND (Pin 6) | GND | Yellow |
TX (Pin 8) | RX | Orange |
RX (Pin 10) | TX | Red |
Boot Log will appear at /dev/ttyUSB0…
## Allow the user to access the USB UART port
## Logout and login to refresh the permissions
sudo usermod -a -G dialout $USER
logout
## Connect to USB UART Console
screen /dev/ttyUSB0 115200
Download the Latest AvaotaOS Release (Ubuntu Noble GNOME) and uncompress it…
wget https://github.com/AvaotaSBC/AvaotaOS/releases/download/0.3.0.4/AvaotaOS-0.3.0.4-noble-gnome-arm64-avaota-a1.img.xz
xz -d AvaotaOS-0.3.0.4-noble-gnome-arm64-avaota-a1.img.xz
Write the .img
file to a MicroSD with Balena Etcher.
Insert the MicroSD into our SBC and Boot AvaotaOS. We’ll see the Boot Log…
read /Image addr=40800000
Kernel addr: 0x40800000
BL31: v2.5(debug):9241004a9
sunxi-arisc driver is starting
ERROR: Error initializing runtime service opteed_fast
Aha! Kernel Boot Address 0x4080_0000 is super important, we’ll use it in a while
Follow these steps to Build and Run NuttX for Arm64 QEMU (Kernel Build)
## Build NuttX Kernel (NuttX Kernel Build)
git clone https://github.com/apache/nuttx
git clone https://github.com/apache/nuttx-apps apps
cd nuttx
tools/configure.sh qemu-armv8a:knsh
make -j
## Build NuttX Apps (NuttX Kernel Build)
make -j export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make -j import
popd
## Boot NuttX on QEMU
qemu-system-aarch64 \
-semihosting \
-cpu cortex-a53 \
-nographic \
-machine virt,virtualization=on,gic-version=3 \
-net none \
-chardev stdio,id=con,mux=on \
-serial chardev:con \
-mon chardev=con,mode=readline \
-kernel ./nuttx
Check that it works…
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
NuttShell (NSH) NuttX-12.8.0
nsh> uname -a
nxposix_spawn_exec: ERROR: exec failed: 2
NuttX 12.8.0 96eb5e7819 Mar 13 2025 15:45:11 arm64 qemu-armv8a
nsh> hello
Hello, World!!
## No worries about `nxposix_spawn_exec`
## To Quit: Press Ctrl-a then x
We’re ready to boot nuttx.bin
on our SBC.
What’s this semihosting business in QEMU?
## Boot NuttX on QEMU, needs Semihosting
qemu-system-aarch64 \
-semihosting ...
NuttX Kernel Build will call out to HostFS Semihosting (pic above) to access NSH Shell and NuttX Apps. We’ll change this for our SBC.
Why start with NuttX Kernel Build? Not NuttX Flat Build?
Our SBC is a mighty monster with Eight Arm64 Cores and plenty of RAM (2 GB). It makes more sense to boot NuttX Kernel Build and run lots of cool powerful NuttX App, thanks to Virtual Memory.
(NuttX Flat Build was created for Simpler Microcontrollers with Limited RAM)
Remember the MicroSD we downloaded? Inside the MicroSD is a 28 MB Linux Kernel, named “Image
”…
$ ls -l /media/$USER/boot
78769 bl31.bin
180233 config-5.15.154-ga464bc4feaff
512 dtb
512 extlinux
27783176 Image
180228 scp.bin
12960 splash.bin
5193581 System.map-5.15.154-ga464bc4feaff
6497300 uInitrd
We replace it with NuttX…
Take the NuttX Kernel nuttx.bin
from the previous section
(Yes the QEMU one)
Overwrite the Image
file by nuttx.bin
…
## Backup and overwrite `Image` on MicroSD
mv \
/media/$USER/boot/Image \
/media/$USER/boot/Image.old
cp \
nuttx.bin \
/media/$USER/boot/Image
## `Image` should be a lot smaller now
## Remember to Unmount and prevent filesystem corruption
ls -l /media/$USER/boot/Image
umount /media/$USER/boot
Insert the MicroSD into our SBC. Boot it…
read /Image addr=40800000
Kernel addr: 0x40800000
BL31: v2.5(debug):9241004a9
sunxi-arisc driver is starting
ERROR: Error initializing runtime service opteed_fast
Nothing happens. We tweak this iteratively, in tiny steps…
Is NuttX actually booting on our SBC?
Let’s print something. UART0 Base Address is here…
A523 User Manual | Page 1839 |
---|---|
Module | Base Address |
UART0 | 0x0250_0000 |
16550 Transmit Register is at Offset 0…
A523 User Manual | Page 1839 |
---|---|
Offset | Register |
0x0000 | UART_THR (Transmit Holding Register) |
0x0004 | UART_DLH (Divisor Latch High Register) |
0x0008 | UART_IIR (Interrupt Identity Register) |
0x000C | UART_LCR (Line Control) |
This says we can Print to UART like so…
// Print `123` to UART0
*(volatile uint8_t *) 0x02500000 = '1';
*(volatile uint8_t *) 0x02500000 = '2';
*(volatile uint8_t *) 0x02500000 = '3';
But we’ll do it in Arm64 Assembly: arm64_head.S
/* Bootloader starts NuttX here */
__start:
add x13, x18, #0x16 /* "MZ": Magic Number for Linux Kernel Header */
b real_start /* Jump to Executable Code */
... /* Omitted: Linux Kernel Header */
/* Executable Code begins here */
/* We print `123` to UART0 */
real_start:
/* Load UART0 Base Address into Register X15 */
mov x15, #0x02500000
/* Load character `1` into Register W16 */
mov w16, #0x31
/* Store the lower byte from Register W16 (`1`) to UART0 Base Address */
strb w16, [x15]
/* Load and Store the lower byte from Register W16 (`2`) to UART0 Base Address */
mov w16, #0x32
strb w16, [x15]
/* Load and Store the lower byte from Register W16 (`3`) to UART0 Base Address */
mov w16, #0x33
strb w16, [x15]
Rebuild NuttX and recopy nuttx.bin
to MicroSD, overwriting the Image
file. NuttX will boot and print 123
! 🎉
read /Image addr=40800000
Kernel addr: 0x40800000
BL31: v2.5(debug):9241004a9
sunxi-arisc driver is starting
ERROR: Error initializing runtime service opteed_fast
123
Indeed NuttX is booting on our SBC, then crashing later. (Ignore the error: opteed_fast)
Why print in Arm64 Assembly? Why not C?
Arm64 Assembly is the very first thing that boots when Bootloader starts NuttX
This happens before anything complicated and crash-prone begins: UART Driver, Memory Management, Task Scheduler, …
The Arm64 Assembly above is Address-Independent Code: It will execute at Any Arm64 Address
Next we move our code and make it Address-Dependent…
NuttX boots a tiny bit on our SBC. Where’s the rest?
Our SBC boots NuttX at a different address from QEMU. We set the Start Address inside NuttX…
read /Image addr=40800000
Kernel addr: 0x40800000
123
Remember the Boot Log from earlier? It says that the SyterKit Bootloader starts NuttX at Address 0x4080_0000
. We set it here: ld-kernel.script
MEMORY {
/* Previously: QEMU boots at 0x4028_0000 */
dram (rwx) : ORIGIN = 0x40800000, LENGTH = 2M
/* Previously: QEMU Paged Memory is at 0x4028_0000 */
pgram (rwx) : ORIGIN = 0x40A00000, LENGTH = 4M /* w/ cache */
/* Why? Because 0x4080_0000 + 2 MB = 0x40A0_0000 */
(Note that Paged Memory Pool shifts down)
Since we changed the Paged Memory Pool (pgram), we update ARCH_PGPOOL_PBASE and VBASE: configs/knsh/defconfig
## Physical Address of Paged Memory Pool
## Previously: QEMU Paged Memory is at 0x4028_0000
CONFIG_ARCH_PGPOOL_PBASE=0x40A00000
## Virtual Address of Paged Memory Pool
## Previously: QEMU Paged Memory is at 0x4028_0000
CONFIG_ARCH_PGPOOL_VBASE=0x40A00000
(Paged Memory Pool shall be dished out as Virtual Memory to NuttX Apps)
NuttX QEMU declares the RAM Size as 128 MB in RAMBANK1_SIZE. We set RAM_SIZE accordingly: configs/knsh/defconfig
## RAM Size is a paltry 128 MB
CONFIG_RAM_SIZE=134217728
(Kinda tiny, but sufficient)
Linux Kernel Header has an incorrect Image Load Offset. Arm64 Bootloaders don’t care, we’ll let it be…
/* Bootloader starts NuttX here, followed by Linux Kernel Header */
__start:
...
/* Image Load Offset from Start of RAM */
/* Boot Address - CONFIG_RAM_START = 0x800000 */
/* But we won't change this, since it's unused */
.quad 0x800000
With these mods, our C Code in NuttX shall boot correctly. FYI: Boot Address also appears on the Onboard LCD…
Our C Code can print to UART now?
To watch the Boot Progress (Sesame Street-style), we can print primitively to UART like this: qemu_boot.c
// 0x0250_0000 is the UART0 Base Address
void arm64_boot_primary_c_routine(void) {
*(volatile uint8_t *) 0x02500000 = 'A';
arm64_chip_boot();
...
void arm64_chip_boot(void) {
*(volatile uint8_t *) 0x02500000 = 'B';
arm64_mmu_init(true); // Init the Memory Mgmt Unit
*(volatile uint8_t *) 0x02500000 = 'C';
arm64_enable_mte(); // Init the Memory Tagging Extension
*(volatile uint8_t *) 0x02500000 = 'D';
qemu_board_initialize(); // Init the Board
*(volatile uint8_t *) 0x02500000 = 'E';
arm64_earlyserialinit(); // Init the Serial Driver
*(volatile uint8_t *) 0x02500000 = 'F';
syslog_rpmsg_init_early(...); // Init the System Logger
*(volatile uint8_t *) 0x02500000 = 'G';
up_perf_init(..); // Init the Performance Counters
Beyond Big Bird: We need the 16550 UART Driver. Based on the A527 UART Doc…
NuttX Boot Code (Arm64 Assembly) will print to UART. We patch it: qemu_lowputc.S
// Base Address and Baud Rate for 16550 UART
#define UART1_BASE_ADDRESS 0x02500000
#define EARLY_UART_PL011_BAUD_RATE 115200
NuttX Boot Code will drop UART Output, unless we wait for UART Ready: qemu_lowputc.S
/* Wait for 16550 UART to be ready to transmit
* xb: Register that contains the UART Base Address
* wt: Scratch register number */
.macro early_uart_ready xb, wt
1:
ldrh \wt, [\xb, #0x14] /* UART_LSR (Line Status Register) */
tst \wt, #0x20 /* Check THRE (TX Holding Register Empty) */
b.eq 1b /* Wait for the UART to be ready (THRE=1) */
.endm
QEMU uses PL011 UART. We switch to 16550 UART: qemu_serial.c
// Switch from PL011 UART (QEMU) to 16550 UART
#include <nuttx/serial/uart_16550.h>
// Enable the 16550 Console UART at Startup
void arm64_earlyserialinit(void) {
// Previously for QEMU: pl011_earlyserialinit
u16550_earlyserialinit();
}
// Ditto but not so early
void arm64_serialinit(void) {
// Previous for QEMU: pl011_serialinit
u16550_serialinit();
}
16550 UART shall be configured: configs/knsh/defconfig
CONFIG_16550_ADDRWIDTH=0
CONFIG_16550_REGINCR=4
CONFIG_16550_UART0=y
CONFIG_16550_UART0_BASE=0x02500000
CONFIG_16550_UART0_CLOCK=198144000
CONFIG_16550_UART0_IRQ=125
CONFIG_16550_UART0_SERIAL_CONSOLE=y
CONFIG_16550_UART=y
CONFIG_16550_WAIT_LCR=y
CONFIG_SERIAL_UART_ARCH_MMIO=y
PL011 UART shall be removed: configs/knsh/defconfig
## Remove PL011 UART from NuttX Config:
## CONFIG_UART1_BASE=0x9000000
## CONFIG_UART1_IRQ=33
## CONFIG_UART1_PL011=y
## CONFIG_UART1_SERIAL_CONSOLE=y
## CONFIG_UART_PL011=y
16550_UART0_CLOCK isn’t quite correct, we’ll settle later. Meanwhile we disable the UART Clock Configuration: uart_16550.c
// We disable the UART Clock Configuration...
static int u16550_setup(FAR struct uart_dev_s *dev) { ...
#ifdef FIX_LATER // We'll fix it later
// Enter DLAB=1
u16550_serialout(priv, UART_LCR_OFFSET, (lcr | UART_LCR_DLAB));
// Omitted: Set the UART Baud Divisor
// ...
// Clear DLAB
u16550_serialout(priv, UART_LCR_OFFSET, lcr);
#endif
Same old drill: Rebuild, recopy and reboot NuttX. We see plenty more debug output…
123
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
AB
OK the repeated rebuilding, recopying and rebooting of NuttX is getting really tiresome. We automate…
What if we could rebuild-recopy-reboot NuttX… In One Single Script?
Thankfully our Avaota-A1 SBC is connected to SDWire MicroSD Multiplexer and Smart Power Plug (pic above). Our Build Script shall do everything for us…
Copy NuttX to MicroSD
Swap the MicroSD from our Test PC to SBC
Power up SBC and boot NuttX
Run OSTest and verify the result
How it looks? Watch the Demo
All this happens in our nifty Build Script: run.sh
## Build NuttX and Apps (NuttX Kernel Build)
make -j
make -j export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make -j import
popd
## Generate the Initial RAM Disk
## Prepare a Padding with 64 KB of zeroes
## Append Padding and Initial RAM Disk to the NuttX Kernel
genromfs -f initrd -d ../apps/bin -V "NuttXBootVol"
head -c 65536 /dev/zero >/tmp/nuttx.pad
cat nuttx.bin /tmp/nuttx.pad initrd \
>Image
## Get the Home Assistant Token
## That we copied from http://localhost:8123/profile/security
## export token=xxxx
. $HOME/home-assistant-token.sh
## Power Off the SBC
curl \
-X POST \
-H "Authorization: Bearer $token" \
-H "Content-Type: application/json" \
-d '{"entity_id": "automation.avaota_power_off"}' \
http://localhost:8123/api/services/automation/trigger
## Copy NuttX Image to MicroSD
## No password needed for sudo, see below
## Change `thinkcentre` to your Test PC
## https://github.com/lupyuen/nuttx-avaota-a1/blob/main/copy-image.sh
scp Image thinkcentre:/tmp/Image
ssh thinkcentre ls -l /tmp/Image
ssh thinkcentre sudo /home/user/copy-image.sh
## Power On the SBC
curl \
-X POST \
-H "Authorization: Bearer $token" \
-H "Content-Type: application/json" \
-d '{"entity_id": "automation.avaota_power_on"}' \
http://localhost:8123/api/services/automation/trigger
## Wait for SBC to finish booting
sleep 30
## Power Off the SBC
curl \
-X POST \
-H "Authorization: Bearer $token" \
-H "Content-Type: application/json" \
-d '{"entity_id": "automation.avaota_power_off"}' \
http://localhost:8123/api/services/automation/trigger
This script assumes that we have…
Installed a Home Assistant Server
(Works fine with Docker)
Added the Smart Power Plug to Google Assistant
“Avaota Power” (pic above)
Installed the Google Assistant SDK for Home Assistant
(So we don’t need Zigbee programming)
Created the Power Automation in Home Assistant
“Avaota Power On” and “Avaota Power Off” (pic below)
What’s copy_image.sh?
This is the script that copies our NuttX Image to MicroSD, via the SDWire MicroSD Multiplexer, explained here…
Can we automate the testing? Including OSTest?
Yep we have an Expect Script that will execute and verify OSTest: run.sh
## Copy NuttX Image to MicroSD
## No password needed for sudo, see below
## Change `thinkcentre` to your Test PC
## https://github.com/lupyuen/nuttx-avaota-a1/blob/main/copy-image.sh
scp Image thinkcentre:/tmp/Image
ssh thinkcentre ls -l /tmp/Image
ssh thinkcentre sudo /home/user/copy-image.sh
## Boot and Test NuttX with OSTest
## https://github.com/lupyuen/nuttx-build-farm/blob/main/avaota.exp
export AVAOTA_SERVER=thinkcentre
pushd $HOME/nuttx-build-farm
expect ./avaota.exp
popd
It’s getting late. Can we get back to NuttX now?
24 Hours is all we need no worries! Earlier we saw NuttX stuck at “AB”…
123
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
AB
Which says that NuttX is stranded inside arm64_mmu_init: qemu_boot.c
// 0x0250_0000 is the UART0 Base Address
void arm64_boot_primary_c_routine(void) {
*(volatile uint8_t *) 0x02500000 = 'A';
arm64_chip_boot();
...
// `AB` means that NuttX is stuck inside arm64_mmu_init()
void arm64_chip_boot(void) {
*(volatile uint8_t *) 0x02500000 = 'B';
arm64_mmu_init(true); // Init the Memory Mgmt Unit
// Stuck above, never came here
*(volatile uint8_t *) 0x02500000 = 'C';
arm64_enable_mte(); // Init the Memory Tagging Extension
What’s arm64_mmu_init?
NuttX calls arm64_mmu_init to start the Arm64 Memory Management Unit (MMU). We add some logs inside: arm64_mmu.c
// Enable Debugging for MMU
#define CONFIG_MMU_ASSERT 1
#define CONFIG_MMU_DEBUG 1
#define trace_printf _info
// We fix the Debug Output, changing `%lux` to `%p`
static void init_xlat_tables(const struct arm_mmu_region *region) {
...
sinfo("mmap: virt %p phys %p size %p\n", virt, phys, size);
// To enable the MMU at Exception Level 1...
static void enable_mmu_el1(unsigned int flags) {
...
// Flush the Cached Data before Enabling MMU
_info("UP_MB");
UP_MB();
// Enable the MMU and Data Cache
_info("Enable the MMU and data cache");
write_sysreg(value | SCTLR_M_BIT | SCTLR_C_BIT, sctlr_el1);
// Ensure that MMU Enable takes effect immediately
_info("UP_ISB");
UP_ISB();
And we enable the logs for Scheduler and Memory Manager: configs/knsh/defconfig
## Enable Logging for Memory Manager
CONFIG_DEBUG_MM=y
CONFIG_DEBUG_MM_ERROR=y
CONFIG_DEBUG_MM_INFO=y
CONFIG_DEBUG_MM_WARN=y
## Enable Logging for Scheduler
CONFIG_DEBUG_SCHED=y
CONFIG_DEBUG_SCHED_ERROR=y
CONFIG_DEBUG_SCHED_INFO=y
CONFIG_DEBUG_SCHED_WARN=y
Ah OK we’re stuck just before Enabling the MMU…
## Init the MMU Page Translation Tables
init_xlat_tables: mmap: virt 0x7000000 phys 0x7000000 size 0x20000000
init_xlat_tables: mmap: virt 0x40000000 phys 0x40000000 size 0x8000000
init_xlat_tables: mmap: virt 0x4010000000 phys 0x4010000000 size 0x10000000
init_xlat_tables: mmap: virt 0x8000000000 phys 0x8000000000 size 0x8000000000
init_xlat_tables: mmap: virt 0x3eff0000 phys 0x3eff0000 size 0x10000
init_xlat_tables: mmap: virt 0x40800000 phys 0x40800000 size 0x2a000
init_xlat_tables: mmap: virt 0x4082a000 phys 0x4082a000 size 0x6000
init_xlat_tables: mmap: virt 0x40830000 phys 0x40830000 size 0x13000
init_xlat_tables: mmap: virt 0x40a00000 phys 0x40a00000 size 0x400000
## Enable the MMU at Exception Level 1
enable_mmu_el1: UP_MB
enable_mmu_el1: Enable the MMU and data cache
Something sus about the above Mystery Addresses, what are they?
Virtual | Physical | Size |
---|---|---|
0x0700_0000 | 0x0700_0000 | 0x2000_0000 |
0x4000_0000 | 0x4000_0000 | 0x0800_0000 |
0x40_1000_0000 | 0x40_1000_0000 | 0x1000_0000 |
0x80_0000_0000 | 0x80_0000_0000 | 0x80_0000_0000 |
0x3EFF_0000 | 0x3EFF_0000 | 0x0001_0000 |
0x4080_0000 | 0x4080_0000 | 0x0002_A000 |
0x4082_A000 | 0x4082_A000 | 0x0000_6000 |
0x4083_0000 | 0x4083_0000 | 0x0001_3000 |
0x40A0_0000 | 0x40A0_0000 | 0x0040_0000 |
Why do we need Arm64 MMU? (Memory Management Unit)
We require MMU for…
Memory Protection: Prevent Applications (and Kernel) from meddling with things (in System Memory) that they’re not supposed to
Virtual Memory: Allow Applications to access chunks of “Imaginary Memory” at Exotic Addresses (0x8000_0000!)
But in reality: They’re System RAM recycled from boring old addresses (like 0x40A0_4000)
If we don’t configure MMU with the correct Memory Map…
NuttX Kernel won’t boot: “Help! I can’t access my Kernel Code and Data!”
NuttX Apps won’t run: “Whoops where’s the App Code and Data that Kernel promised?”
Arm64 MMU won’t turn on. Maybe our Memory Map is incorrect?
We verify our A527 Memory Map (pic above)
A523 User Manual | Page 40 |
---|---|
Module | Address |
Boot ROM & SRAM | 0x0000_0000 to … |
PCIE | 0x2000_0000 to 0x2FFF_FFFF |
DRAM | 0x4000_0000 to … |
How does this compare with NuttX? We do extra MMU Logging: arm64_mmu.c
// Print the Names of the MMU Memory Regions
static void init_xlat_tables(const struct arm_mmu_region *region) { ...
_info("name=%s\n", region->name);
sinfo("mmap: virt %p phys %p size %p\n", virt, phys, size);
Ah much clearer! Now we see the Names of Memory Regions for the MMU…
Name | Physical | Size |
---|---|---|
DEVICE_REGION | 0x0700_0000 | 0x2000_0000 |
DRAM0_S0 | 0x4000_0000 | 0x0800_0000 |
PCI_CFG | 0x40_1000_0000 | 0x1000_0000 |
PCI_MEM | 0x80_0000_0000 | 0x80_0000_0000 |
PCI_IO | 0x3EFF_0000 | 0x0001_0000 |
nx_code | 0x4080_0000 | 0x0002_A000 |
nx_rodata | 0x4082_A000 | 0x0000_6000 |
nx_data | 0x4083_0000 | 0x0001_3000 |
nx_pgpool | 0x40A0_0000 | 0x0040_0000 |
Two Tweaks…
DEVICE_REGION: This says I/O Memory Space ends at 0x2700_0000. Based on the earlier A527 Memory Map, we extend this to 0x4000_0000 (1 GB): qemu/chip.h
// Fix the I/O Memory Space: Base Address and Size
#define CONFIG_DEVICEIO_BASEADDR 0x00000000
#define CONFIG_DEVICEIO_SIZE MB(1024)
// We don't need PCI, for now
// #define CONFIG_PCI_CFG_BASEADDR 0x4010000000
// #define CONFIG_PCI_CFG_SIZE MB(256)
// #define CONFIG_PCI_MEM_BASEADDR 0x8000000000
// #define CONFIG_PCI_MEM_SIZE GB(512)
// #define CONFIG_PCI_IO_BASEADDR 0x3eff0000
// #define CONFIG_PCI_IO_SIZE KB(64)
PCI: We remove these for now: qemu_boot.c
static const struct arm_mmu_region g_mmu_regions[] = {
...
// We don't need PCI, for now
// MMU_REGION_FLAT_ENTRY("PCI_CFG", ...
// MMU_REGION_FLAT_ENTRY("PCI_MEM", ...
// MMU_REGION_FLAT_ENTRY("PCI_IO", ...
The rest are hunky dory…
DRAM0_S0 says that RAM Address Space ends at 0x4800_0000 (128 MB)
(Kinda small, but sufficient for now)
nx_code (0x4080_0000): Kernel Code begins here
nx_rodata (0x4082_A000): Read-Only Data for Kernel
nx_data (0x4083_0000): Read-Write Data for Kernel
nx_pgpool (0x40A0_0000): Remember the Paged Memory Pool? This shall be dished out as Virtual Memory to NuttX Apps
We rebuild, recopy, reboot NuttX. Our Memory Map looks much better now…
Name | Physical | Size |
---|---|---|
DEVICE_REGION | 0x0000_0000 | 0x4000_0000 |
DRAM0_S0 | 0x4000_0000 | 0x0800_0000 |
nx_code | 0x4080_0000 | 0x0002_A000 |
nx_rodata | 0x4082_A000 | 0x0000_6000 |
nx_data | 0x4083_0000 | 0x0001_3000 |
nx_pgpool | 0x40A0_0000 | 0x0040_0000 |
Though it crashes elsewhere…
Why is NuttX failing with an Undefined Instruction?
gic_validate_dist_version: No GIC version detect
arm64_gic_initialize: no distributor detected, giving up ret=-19
...
nx_start_application: Starting init task: /system/bin/init
arm64_el1_undef: Undefined instruction at 0x408276e4, dump:
Assertion failed panic: at file: common/arm64_fatal.c:572
Yeah this failure is totally misleading. Real Culprit: NuttX couldn’t Init the GIC…
No GIC version detect
No distributor detected, giving up
What’s this GIC?
It’s the Arm64 Generic Interrupt Controller (GIC), version 3. GIC shall…
Receive Input / Output Interrupts
(Like keypresses)
Forward them to a CPU Core for processing
(Works like RISC-V PLIC)
GIC is here…
A523 User Manual | Page 263 |
---|---|
Module | Base Address |
GIC | 0x0340_0000 |
Which has these GIC Registers inside, handling 8 Arm64 Cores…
A523 User Manual | Page 264 |
---|---|
Offset | Register |
0x00_0000 | GICD_CTLR (Distributor Control Register) |
0x06_0000 | GICR_CTLR_C0 (Redistributor Control Register, Core 0) |
0x08_0000 | GICR_CTLR_C1 (Ditto, Core 1) |
0x0A_0000 | GICR_CTLR_C2 (Ditto, Core 2) |
0x0C_0000 | GICR_CTLR_C3 (Ditto, Core 3) |
0x0E_0000 | GICR_CTLR_C4 (Ditto, Core 4) |
0x10_0000 | GICR_CTLR_C5 (Ditto, Core 5) |
0x12_0000 | GICR_CTLR_C6 (Ditto, Core 6) |
0x14_0000 | GICR_CTLR_C7 (Ditto, Core 7) |
0x16_0000 | GICDA_CTLR (Distributor Control Register A) |
Based on the above, we set the Addresses of GICD and GICR (Distributor / Redistributor): qemu/chip.h
// Base Address of GIC Distributor and Redistributor
#define CONFIG_GICD_BASE 0x3400000
#define CONFIG_GICR_BASE 0x3460000
// Spaced 0x20000 bytes per Arm64 Core
#define CONFIG_GICR_OFFSET 0x20000
Remember to Disable Memory Manager Logging. NuttX GIC Driver starts correctly and complains no more!
## SPI = Physical Interrupt Signal (not the typical SPI)
gic_validate_dist_version:
GICv3 version detect
GICD_TYPER = 0x7b0408
256 SPIs implemented
We’ll call GIC to handle UART Interrupts. Before that: We need NSH Shell…
Are we done yet?
For a Simple NuttX Port (Flat Build): Congrats, just fix the UART Interrupt and we’re done!
However we’re doing NuttX Kernel Build. Which needs more work…
nx_start_application:
Starting init task: /system/bin/init
arm64_el1_undef:
Undefined instruction at 0x408274a4
Assertion failed panic:
common/arm64_fatal.c:572
What’s /system/bin/init? Why is it failing?
/system/bin/init is NSH Shell. NuttX Kernel Build will load NuttX Apps from a Local Filesystem, and it’s missing right now. (NuttX Flat Build will bind binary Apps directly into Kernel)
To solve this: We bundle the NuttX Apps together into a ROMFS Filesystem…
## Generate the Initial RAM Disk
genromfs \
-f initrd \
-d ../apps/bin \
-V "NuttXBootVol"
Then we package NuttX Kernel + NuttX Apps into a NuttX Image…
## Prepare a Padding with 64 KB of zeroes
## Append Padding and Initial RAM Disk to the NuttX Kernel
head -c 65536 /dev/zero >/tmp/nuttx.pad
cat nuttx.bin /tmp/nuttx.pad initrd \
>Image
When NuttX Boots: It will…
Locate the ROMFS Filesystem in memory
Copy it to the designated Memory Region
Mount it as a RAM Disk
Allowing NuttX Kernel to start NSH Shell
(And other NuttX Apps)
How? See this…
NSH Prompt still missing? It won’t appear until we handle the UART Interrupt…
One Last Thing: Settle the UART Interrupt and we’re done!
A523 User Manual | Page 256 |
---|---|
Interrupt Number | Interrupt Source |
34 | UART0 |
This is how we set the UART0 Interrupt and watch for keypresses: configs/knsh/defconfig
## Set the UART0 Interrupt to 34
CONFIG_16550_UART0_IRQ=34
To Wrap Up: We Disable Logging for Memory Manager and Scheduler. And Disable MMU Debugging.
NSH Prompt finally appears and OSTest completes successfully. Our NuttX Porting is complete yay!
NuttShell (NSH) NuttX-12.4.0
nsh> uname -a
NuttX 12.4.0 6c5c1a5f9f-dirty Mar 8 2025 21:57:02 arm64 qemu-armv8a
nsh> ostest
...
user_main: Exiting
ostest_main: Exiting with status 0
NSH Prompt won’t appear if UART Interrupt is disabled?
That’s because NSH runs as a NuttX App in User Space. When NSH Shell prints this…
NuttShell (NSH) NuttX-12.4.0
nsh>
It calls the Serial Driver. Which will wait for a UART Interrupt to signal that the Transmit Buffer is empty and available.
Thus if UART Interrupt is disabled, nothing gets printed in NuttX Apps. (Explained here)
NuttX might run OK on Radxa Cubie A5E (Allwinner T527)
Right now we’re upstreaming Avatoa-A1 SBC to NuttX Mainline…
We’re seeking volunteers to build NuttX Drivers for Avaota-A1 (GPIO, SPI, I2C, MIPI CSI / DSI, Ethernet, WiFi, …) Please lemme know, maybe we can draw something on the Onboard LCD!
Special Thanks to My Sponsors for supporting my writing. Your support means so much to me 🙏
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…
To boot NuttX on Avatoa-A1: We may download Image
from here…
Or follow these steps to compile our (Work-In-Progress) NuttX for Avaota-A1: run.sh
## Download Source Code for NuttX and Apps
git clone https://github.com/lupyuen2/wip-nuttx nuttx --branch avaota2
git clone https://github.com/lupyuen2/wip-nuttx-apps apps --branch avaota2
cd nuttx
## Build NuttX and Apps (NuttX Kernel Build)
tools/configure.sh avaota-a1:nsh
make -j
make -j export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make -j import
popd
## Generate the Initial RAM Disk
## Prepare a Padding with 64 KB of zeroes
## Append Padding and Initial RAM Disk to the NuttX Kernel
genromfs -f initrd -d ../apps/bin -V "NuttXBootVol"
head -c 65536 /dev/zero >/tmp/nuttx.pad
cat nuttx.bin /tmp/nuttx.pad initrd \
>Image
Read on to boot the NuttX Image on our SBC…
Earlier we built NuttX for Avaota-A1 and created the Image
file, containing the NuttX Kernel + NuttX Apps. Let’s boot it on MicroSD…
Prepare the AvaotaOS MicroSD…
Copy the NuttX Image to MicroSD…
## Copy NuttX Image to AvaotaOS MicroSD
## Overwrite the `Image` file
mv \
/media/$USER/boot/Image \
/media/$USER/boot/Image.old
cp \
Image \
/media/$USER/boot/Image
## Unmount and boot it on Avaota-A1
ls -l /media/$USER/boot/Image
umount /media/$USER/boot
Boot the MicroSD on our SBC
We can automate the last two steps with a MicroSD Multiplexer and Smart Power Plug: run.sh
## Get the Home Assistant Token
## That we copied from http://localhost:8123/profile/security
## export token=xxxx
. $HOME/home-assistant-token.sh
## Power Off the SBC
curl \
-X POST \
-H "Authorization: Bearer $token" \
-H "Content-Type: application/json" \
-d '{"entity_id": "automation.avaota_power_off"}' \
http://localhost:8123/api/services/automation/trigger
## Copy NuttX Image to MicroSD
## No password needed for sudo, see below
## Change `thinkcentre` to your Test PC
## https://github.com/lupyuen/nuttx-avaota-a1/blob/main/copy-image.sh
scp Image thinkcentre:/tmp/Image
ssh thinkcentre ls -l /tmp/Image
ssh thinkcentre sudo /home/user/copy-image.sh
## Power On the SBC
curl \
-X POST \
-H "Authorization: Bearer $token" \
-H "Content-Type: application/json" \
-d '{"entity_id": "automation.avaota_power_on"}' \
http://localhost:8123/api/services/automation/trigger
## Wait for SBC to finish testing
echo Press Enter to Power Off
read
## Power Off the SBC
curl \
-X POST \
-H "Authorization: Bearer $token" \
-H "Content-Type: application/json" \
-d '{"entity_id": "automation.avaota_power_off"}' \
http://localhost:8123/api/services/automation/trigger
Or to Automate Everything including OSTest: run.sh
## Copy NuttX Image to MicroSD
## No password needed for sudo, see below
## Change `thinkcentre` to your Test PC
## https://github.com/lupyuen/nuttx-avaota-a1/blob/main/copy-image.sh
scp Image thinkcentre:/tmp/Image
ssh thinkcentre ls -l /tmp/Image
ssh thinkcentre sudo /home/user/copy-image.sh
## Boot and Test NuttX with OSTest
## https://github.com/lupyuen/nuttx-build-farm/blob/main/avaota.exp
export AVAOTA_SERVER=thinkcentre
pushd $HOME/nuttx-build-farm
expect ./avaota.exp
popd
(copy-image.sh is explained here)
NuttX boots to NSH Shell. And passes OSTest yay!
NOTICE: BL31: v2.5(debug):9241004a9
NOTICE: BL31: Built : 13:37:46, Nov 16 2023
NOTICE: BL31: No DTB found.
NOTICE: [SCP] :wait arisc ready....
NOTICE: [SCP] :arisc version: []
NOTICE: [SCP] :arisc startup ready
NOTICE: [SCP] :arisc startup notify message feedback
NOTICE: [SCP] :sunxi-arisc driver is starting
ERROR: Error initializing runtime service opteed_fast
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
NuttShell (NSH) NuttX-12.4.0
nsh> uname -a
NuttX 12.4.0 42c0ed4a89 Mar 13 2025 09:13:56 arm64 avaota-a1
nsh> ls -l /dev
/dev:
crw-rw-rw- 0 console
crw-rw-rw- 0 null
brw-rw-rw- 16777216 ram0
crw-rw-rw- 0 ttyS0
crw-rw-rw- 0 zero
nsh> ps
PID GROUP PRI POLICY TYPE NPX STATE EVENT SIGMASK STACK USED FILLED COMMAND
0 0 0 FIFO Kthread - Ready 0000000000000000 0008176 0000928 11.3% Idle_Task
1 0 192 RR Kthread - Waiting Semaphore 0000000000000000 0008112 0000992 12.2% hpwork 0x40833568 0x408335b8
2 0 100 RR Kthread - Waiting Semaphore 0000000000000000 0008112 0000992 12.2% lpwork 0x408334e8 0x40833538
4 4 100 RR Task - Running 0000000000000000 0008128 0002192 26.9% /system/bin/init
nsh> free
total used free maxused maxfree nused nfree name
125542400 33880 125508520 53032 125484976 58 5 Kmem
4194304 245760 3948544 3948544 Page
nsh> hello
Hello, World!!
nsh> getprime
Set thread priority to 10
Set thread policy to SCHED_RR
Start thread #0
thread #0 started, looking for primes < 10000, doing 10 run(s)
thread #0 finished, found 1230 primes, last one was 9973
Done
getprime took 162 msec
nsh> ostest
...
Final memory usage:
VARIABLE BEFORE AFTER
======== ======== ========
arena a000 26000
ordblks 2 4
mxordblk 6ff8 1aff8
uordblks 27e8 6700
fordblks 7818 1f900
user_main: Exiting
ostest_main: Exiting with status 0
nsh>
In this article we ported NuttX QEMU Arm64 (Kernel Build) iteratively to Avaota-A1. Up Next: Upstreaming our code to NuttX Mainline!
Here’s how we copy-n-pasted our Modified Files into a proper NuttX Arch (Allwinner A527) and NuttX Board (Avaota-A1)
Upstreaming becomes lotsa copypasta…
We create a Staging PR in our own repo…
Dump the list of Modified Files…
## Change this to our Staging PR
$ pr=https://github.com/lupyuen2/wip-pinephone-nuttx/pull/99
$ curl -L $pr.diff \
| grep "diff --git" \
| sort \
| cut -d" " -f3 \
| cut -c3-
## Here are the Modified Files for our PR
Documentation/platforms/arm64/a527/boards/avaota-a1/avaota-a1.jpg
Documentation/platforms/arm64/a527/boards/avaota-a1/index.rst
Documentation/platforms/arm64/a527/index.rst
arch/arm64/Kconfig
arch/arm64/include/a527/chip.h
arch/arm64/include/a527/irq.h
arch/arm64/src/a527/CMakeLists.txt
arch/arm64/src/a527/Kconfig
arch/arm64/src/a527/Make.defs
arch/arm64/src/a527/a527_boot.c
arch/arm64/src/a527/a527_boot.h
arch/arm64/src/a527/a527_initialize.c
arch/arm64/src/a527/a527_lowputc.S
arch/arm64/src/a527/a527_serial.c
arch/arm64/src/a527/a527_textheap.c
arch/arm64/src/a527/a527_timer.c
arch/arm64/src/a527/chip.h
boards/Kconfig
boards/arm64/a527/avaota-a1/CMakeLists.txt
boards/arm64/a527/avaota-a1/Kconfig
boards/arm64/a527/avaota-a1/configs/nsh/defconfig
boards/arm64/a527/avaota-a1/include/board.h
boards/arm64/a527/avaota-a1/include/board_memorymap.h
boards/arm64/a527/avaota-a1/scripts/Make.defs
boards/arm64/a527/avaota-a1/scripts/gnu-elf.ld
boards/arm64/a527/avaota-a1/scripts/ld.script
boards/arm64/a527/avaota-a1/src/CMakeLists.txt
boards/arm64/a527/avaota-a1/src/Makefile
boards/arm64/a527/avaota-a1/src/a527_appinit.c
boards/arm64/a527/avaota-a1/src/a527_boardinit.c
boards/arm64/a527/avaota-a1/src/a527_bringup.c
boards/arm64/a527/avaota-a1/src/a527_power.c
boards/arm64/a527/avaota-a1/src/avaota-a1.h
Check nxstyle on the Modified Files…
## Run nxstyle on the Modified Files
nxstyle Documentation/platforms/arm64/a527/boards/avaota-a1/avaota-a1.jpg
nxstyle Documentation/platforms/arm64/a527/boards/avaota-a1/index.rst
nxstyle Documentation/platforms/arm64/a527/index.rst
nxstyle arch/arm64/Kconfig
nxstyle arch/arm64/include/a527/chip.h
nxstyle arch/arm64/include/a527/irq.h
nxstyle arch/arm64/src/a527/CMakeLists.txt
nxstyle arch/arm64/src/a527/Kconfig
nxstyle arch/arm64/src/a527/Make.defs
nxstyle arch/arm64/src/a527/a527_boot.c
nxstyle arch/arm64/src/a527/a527_boot.h
nxstyle arch/arm64/src/a527/a527_initialize.c
nxstyle arch/arm64/src/a527/a527_lowputc.S
nxstyle arch/arm64/src/a527/a527_serial.c
nxstyle arch/arm64/src/a527/a527_textheap.c
nxstyle arch/arm64/src/a527/a527_timer.c
nxstyle arch/arm64/src/a527/chip.h
nxstyle boards/Kconfig
nxstyle boards/arm64/a527/avaota-a1/CMakeLists.txt
nxstyle boards/arm64/a527/avaota-a1/Kconfig
nxstyle boards/arm64/a527/avaota-a1/configs/nsh/defconfig
nxstyle boards/arm64/a527/avaota-a1/include/board.h
nxstyle boards/arm64/a527/avaota-a1/include/board_memorymap.h
nxstyle boards/arm64/a527/avaota-a1/scripts/Make.defs
nxstyle boards/arm64/a527/avaota-a1/scripts/gnu-elf.ld
nxstyle boards/arm64/a527/avaota-a1/scripts/ld.script
nxstyle boards/arm64/a527/avaota-a1/src/CMakeLists.txt
nxstyle boards/arm64/a527/avaota-a1/src/Makefile
nxstyle boards/arm64/a527/avaota-a1/src/a527_appinit.c
nxstyle boards/arm64/a527/avaota-a1/src/a527_boardinit.c
nxstyle boards/arm64/a527/avaota-a1/src/a527_bringup.c
nxstyle boards/arm64/a527/avaota-a1/src/a527_power.c
nxstyle boards/arm64/a527/avaota-a1/src/avaota-a1.h
Copy the Arch Files into the Arch Pull Request
“arch/arm64/a527: Add support for Allwinner A527 SoC”
## Download the Branch for Avaota Arch (initially empty)
pushd /tmp
git clone https://github.com/lupyuen2/wip-nuttx avaota-arch --branch avaota-arch
popd
## Copy the Arch Files from src to dest
function copy_files() {
src=.
dest=/tmp/avaota-arch
for file in \
Documentation/platforms/arm64/a527/index.rst \
arch/arm64/Kconfig \
arch/arm64/include/a527/chip.h \
arch/arm64/include/a527/irq.h \
arch/arm64/src/a527/CMakeLists.txt \
arch/arm64/src/a527/Kconfig \
arch/arm64/src/a527/Make.defs \
arch/arm64/src/a527/a527_boot.c \
arch/arm64/src/a527/a527_boot.h \
arch/arm64/src/a527/a527_initialize.c \
arch/arm64/src/a527/a527_lowputc.S \
arch/arm64/src/a527/a527_serial.c \
arch/arm64/src/a527/a527_textheap.c \
arch/arm64/src/a527/a527_timer.c \
arch/arm64/src/a527/chip.h \
do
src_file=$src/$file
dest_file=$dest/$file
dest_dir=$(dirname -- "$dest_file")
set -x
mkdir -p $dest_dir
cp $src_file $dest_file
set +x
done
}
## Copy and commit /tmp/avaota-arch
## Remove the "Supported Boards" (toctree) from Arch Doc
copy_files
code /tmp/avaota-arch
Copy the Board Files into the Board Pull Request
## Download the Branch for Avaota Board (initially empty)
pushd /tmp
git clone https://github.com/lupyuen2/wip-nuttx avaota-board --branch avaota-board
popd
## Copy the Board Files from src to dest
## Copy the Arch Doc again because we restored the "Supported Boards"
function copy_files() {
src=.
dest=/tmp/avaota-board
for file in \
Documentation/platforms/arm64/a527/index.rst \
Documentation/platforms/arm64/a527/boards/avaota-a1/avaota-a1.jpg \
Documentation/platforms/arm64/a527/boards/avaota-a1/index.rst \
boards/Kconfig \
boards/arm64/a527/avaota-a1/CMakeLists.txt \
boards/arm64/a527/avaota-a1/Kconfig \
boards/arm64/a527/avaota-a1/configs/nsh/defconfig \
boards/arm64/a527/avaota-a1/include/board.h \
boards/arm64/a527/avaota-a1/include/board_memorymap.h \
boards/arm64/a527/avaota-a1/scripts/Make.defs \
boards/arm64/a527/avaota-a1/scripts/gnu-elf.ld \
boards/arm64/a527/avaota-a1/scripts/ld.script \
boards/arm64/a527/avaota-a1/src/CMakeLists.txt \
boards/arm64/a527/avaota-a1/src/Makefile \
boards/arm64/a527/avaota-a1/src/a527_appinit.c \
boards/arm64/a527/avaota-a1/src/a527_boardinit.c \
boards/arm64/a527/avaota-a1/src/a527_bringup.c \
boards/arm64/a527/avaota-a1/src/a527_power.c \
boards/arm64/a527/avaota-a1/src/avaota-a1.h \
do
src_file=$src/$file
dest_file=$dest/$file
dest_dir=$(dirname -- "$dest_file")
set -x
mkdir -p $dest_dir
cp $src_file $dest_file
set +x
done
}
## Copy and commit /tmp/avaota-board
copy_files
code /tmp/avaota-board
Remember to create Two Commits Per PR: One Commit for Code, Another Commit for Docs
Need to Squash the Commits (or amend them), but another Code or Doc Commit is stuck in between?
Try Reordering the Commits to the top, before squashing or amending.
We’re finally ready to Submit our Pull Requests!
Let’s make our Tweak-Build-Test Cycle quicker for NuttX. We use SDWire MicroSD Multiplexer (pic above) to flip our MicroSD between Test PC and SBC…
SDWire needs Plenty of Sudo Passwords to flip the multiplexer, mount the filesystem, copy to MicroSD. We make it Sudo Password-Less with visudo…
Wrap all the Sudo Commands into a script: copy-image.sh
## Create a Bash Script: copy-image.sh
## Containing these commands...
set -e ## Exit when any command fails
set -x ## Echo commands
whoami ## I am root!
## Copy /tmp/Image to MicroSD
sd-mux-ctrl --device-serial=sd-wire_02-09 --ts
sleep 5
mkdir -p /tmp/sda1
mount /dev/sda1 /tmp/sda1
cp /tmp/Image /tmp/sda1/
ls -l /tmp/sda1
## Unmount MicroSD and flip it to the Test Device (Avaota-A1)
umount /tmp/sda1
sd-mux-ctrl --device-serial=sd-wire_02-09 --dut
Configure visudo so that our script will run as Sudo Without Password…
## Make our script executable
## Start the Sudoers Editor
chmod +x /home/user/copy-image.sh
sudo visudo
## Add this line:
user ALL=(ALL) NOPASSWD: /home/user/copy-image.sh
Then we can trigger our script remotely via SSH, Without Sudo Password: run.sh
## Copy NuttX Image to MicroSD
## No password needed for sudo yay!
scp Image thinkcentre:/tmp/Image
ssh thinkcentre \
ls -l /tmp/Image
ssh thinkcentre \
sudo /home/user/copy-image.sh
Everything goes into our Build Script for NuttX
How it looks? Watch the Demo
(Actually we could allow anyone in the world to Remotely Build and Test NuttX on our Avaota-A1 SBC hmmm…)
Earlier we talked about the ROMFS Filesystem for NuttX Apps (Initial RAM Disk, pic above)
This section explains how we implemented the NuttX Apps Filesystem…
After this implementation, /system/bin/init (NSH Shell) shall start successfully…
qemu_bringup:
mount_ramdisk:
nx_start_application: ret=0
nx_start_application: Starting init task: /system/bin/init
nxtask_activate: /system/bin/init pid=4,TCB=0x408469f0
nxtask_exit: AppBringUp pid=3,TCB=0x40846190
board_app_initialize:
nx_start: CPU0: Beginning Idle Loop
QEMU uses Semihosting and HostFS to access the NuttX Apps Filesystem. We change to ROMFS: configs/knsh/defconfig
## We added ROMFS...
CONFIG_BOARDCTL_ROMDISK=y
CONFIG_BOARD_LATE_INITIALIZE=y
CONFIG_INIT_MOUNT_TARGET="/system/bin"
## And removed Semihosting HostFS...
## CONFIG_FS_HOSTFS=y
## CONFIG_ARM64_SEMIHOSTING_HOSTFS=y
## CONFIG_ARM64_SEMIHOSTING_HOSTFS_CACHE_COHERENCE=y
## CONFIG_INIT_MOUNT_DATA="fs=../apps"
## CONFIG_INIT_MOUNT_FSTYPE="hostfs"
## CONFIG_INIT_MOUNT_SOURCE=""
## CONFIG_INIT_MOUNT_TARGET="/system"
BOARD_LATE_INITIALIZE is needed because we’ll Mount the ROMFS Filesystem inside qemu_bringup. (See below)
We reserve 16 MB of RAM for the ROMFS Filesystem that will host the NuttX Apps: ld-kernel.script
/* Linker Script: We added the RAM Disk (16 MB) */
MEMORY {
dram (rwx) : ORIGIN = 0x40800000, LENGTH = 2M
pgram (rwx) : ORIGIN = 0x40A00000, LENGTH = 4M /* w/ cache */
ramdisk (rwx) : ORIGIN = 0x40E00000, LENGTH = 16M /* w/ cache */
}
/* We'll reference these in our code */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size = LENGTH(ramdisk);
__ramdisk_end = ORIGIN(ramdisk) + LENGTH(ramdisk);
At Startup: We mount the ROMFS Filesystem (inside RAM) as /dev/ram0: qemu_bringup.c
// At NuttX Startup...
int qemu_bringup(void) {
// We Mount the RAM Disk
mount_ramdisk();
...
}
// Mount a RAM Disk defined in ld.script to /dev/ramX. The RAM Disk
// contains a ROMFS filesystem with applications that can be spawned at
// runtime.
static int mount_ramdisk(void) {
struct boardioc_romdisk_s desc;
desc.minor = RAMDISK_DEVICE_MINOR;
desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
desc.sectsize = SECTORSIZE;
desc.image = __ramdisk_start;
int ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);
if (ret < 0) {
syslog(LOG_ERR, "Ramdisk register failed: %s\n", strerror(errno));
syslog(LOG_ERR, "Ramdisk mountpoint /dev/ram%d\n",RAMDISK_DEVICE_MINOR);
syslog(LOG_ERR, "Ramdisk length %lu, origin %lx\n", (ssize_t)__ramdisk_size, (uintptr_t)__ramdisk_start);
}
return ret;
}
// RAM Disk Definition
#define SECTORSIZE 512
#define NSECTORS(b) (((b) + SECTORSIZE - 1) / SECTORSIZE)
#define RAMDISK_DEVICE_MINOR 0
But Before That: We safely copy the ROMFS Filesystem (Initial RAM Disk) from the NuttX Image into the ramdisk
Memory Region…
This happens just after Bootloader starts NuttX: qemu_boot.c
// Needed for the `aligned_data` macro
#include <nuttx/compiler.h>
// Just after Bootloader has started NuttX...
void arm64_chip_boot(void) {
// We copy the RAM Disk
qemu_copy_ramdisk();
// Omitted: Other initialisation (MMU, ...)
arm64_mmu_init(true);
...
}
// Copy the RAM Disk from NuttX Image to RAM Disk Region.
static void qemu_copy_ramdisk(void) {
const uint8_t aligned_data(8) header[8] = "-rom1fs-";
const uint8_t *limit = (uint8_t *)g_idle_topstack + (256 * 1024);
uint8_t *ramdisk_addr = NULL;
uint8_t *addr;
uint32_t size;
// After Idle Stack Top, search for "-rom1fs-". This is the RAM Disk Address.
// Limit search to 256 KB after Idle Stack Top.
for (addr = g_idle_topstack; addr < limit; addr += 8) {
if (memcmp(addr, header, sizeof(header)) == 0) {
ramdisk_addr = addr;
break;
}
}
// Stop if RAM Disk is missing
if (ramdisk_addr == NULL) {
_err("Missing RAM Disk. Check the initrd padding.");
PANIC();
}
// Read the Filesystem Size from the next 4 bytes (Big Endian)
size = (ramdisk_addr[8] << 24) + (ramdisk_addr[9] << 16) +
(ramdisk_addr[10] << 8) + ramdisk_addr[11] + 0x1f0;
// Filesystem Size must be less than RAM Disk Memory Region
if (size > (size_t)__ramdisk_size) {
_err("RAM Disk Region too small. Increase by %lu bytes.\n", size - (size_t)__ramdisk_size);
PANIC();
}
// Copy the RAM Disk from NuttX Image to RAM Disk Region.
// __ramdisk_start overlaps with ramdisk_addr + size.
qemu_copy_overlap(__ramdisk_start, ramdisk_addr, size);
}
// Copy an overlapping memory region. dest overlaps with src + count.
static void qemu_copy_overlap(uint8_t *dest, const uint8_t *src, size_t count) {
uint8_t *d = dest + count - 1;
const uint8_t *s = src + count - 1;
if (dest <= src) { _err("dest and src should overlap"); PANIC(); }
while (count--) {
volatile uint8_t c = *s; // Prevent compiler optimization
*d = c;
d--;
s--;
}
}
// RAM Disk Region is defined in Linker Script
extern uint8_t __ramdisk_start[];
extern uint8_t __ramdisk_size[];
Why the aligned addresses?
// Header is aligned to 8 bytes
const uint8_t
aligned_data(8) header[8] =
"-rom1fs-";
// Address is also aligned to 8 bytes
for (
addr = g_idle_topstack;
addr < limit;
addr += 8
) {
// Otherwise this will hit Alignment Fault
memcmp(addr, header, sizeof(header));
...
}
We align our Memory Accesses to 8 Bytes. Otherwise we’ll hit an Alignment Fault…
## Alignment Fault at `memcmp(addr, header, sizeof(header))`
default_fatal_handler:
(IFSC/DFSC) for Data/Instruction aborts:
alignment fault
(Strangely: This Alignment isn’t needed for RISC-V)
In this article, we took NuttX for Arm64 QEMU Kernel Build (qemu-armv8a:knsh) and changed it slightly for Avaota-A1 SBC. To help our PR Reviewers: This section explains the Modified Code in our Pull Request…
Only Seven Files were modified from QEMU NuttX. All other files were simply copied and renamed, from QEMU NuttX to Avaota-A1. (Pic above)
arch/arm64/include/a527/chip.h
We define the I/O Memory Space…
// I/O Memory Space
#define CONFIG_DEVICEIO_BASEADDR 0x00000000
#define CONFIG_DEVICEIO_SIZE MB(1024)
// Kernel Boot Address from SBC Bootloader
#define CONFIG_LOAD_BASE 0x40800000
Based on the A527 Memory Map…
A523 User Manual | Page 40 |
---|---|
Module | Address |
Boot ROM & SRAM | 0x0000_0000 to … |
PCIE | 0x2000_0000 to 0x2FFF_FFFF |
DRAM | 0x4000_0000 to … |
arch/arm64/include/a527/chip.h
We set the GIC Base Addresses…
// GICD and GICD Base Addresses
#define CONFIG_GICD_BASE 0x3400000
#define CONFIG_GICR_BASE 0x3460000
Based on the GIC Doc…
A523 User Manual | Page 263 |
---|---|
Module | Base Address |
GIC | 0x0340_0000 |
And GIC Registers…
A523 User Manual | Page 264 |
---|---|
Offset | Register |
0x00_0000 | GICD_CTLR (Distributor Control Register) |
0x06_0000 | GICR_CTLR_C0 (Redistributor Control Register, Core 0) |
0x08_0000 | GICR_CTLR_C1 (Ditto, Core 1) |
0x0A_0000 | GICR_CTLR_C2 (Ditto, Core 2) |
0x0C_0000 | GICR_CTLR_C3 (Ditto, Core 3) |
0x0E_0000 | GICR_CTLR_C4 (Ditto, Core 4) |
0x10_0000 | GICR_CTLR_C5 (Ditto, Core 5) |
0x12_0000 | GICR_CTLR_C6 (Ditto, Core 6) |
0x14_0000 | GICR_CTLR_C7 (Ditto, Core 7) |
0x16_0000 | GICDA_CTLR (Distributor Control Register A) |
boards/arm64/a527/avaota-a1/configs/nsh/defconfig
We set the UART0 Interrupt…
## Set the UART0 Interrupt to 34
CONFIG_16550_UART0_IRQ=34
Based on the A527 Doc…
A523 User Manual | Page 256 |
---|---|
Interrupt Number | Interrupt Source |
34 | UART0 |
arch/arm64/src/a527/a527_lowputc.S
We updated the Arm64 Boot Code for 16550 UART Driver…
We modified the UART Base Address…
// Base Address for 16550 UART
#define UART0_BASE_ADDRESS 0x02500000
QEMU was using PL011 UART. We fixed this for 16550 UART, to Wait for UART Ready (derived from NuttX A64)
/* Wait for 16550 UART to be ready to transmit
* xb: Register that contains the UART Base Address
* wt: Scratch register number */
.macro early_uart_ready xb, wt
1:
ldrh \wt, [\xb, #0x14] /* UART_LSR (Line Status Register) */
tst \wt, #0x20 /* Check THRE (TX Holding Register Empty) */
b.eq 1b /* Wait for the UART to be ready (THRE=1) */
.endm
UART Base Address came from the A527 Doc…
A523 User Manual | Page 1839 |
---|---|
Module | Base Address |
UART0 | 0x0250_0000 |
With these UART Registers…
A523 User Manual | Page 1839 |
---|---|
Offset | Register |
0x0000 | UART_THR (Transmit Holding Register) |
0x0004 | UART_DLH (Divisor Latch High Register) |
0x0008 | UART_IIR (Interrupt Identity Register) |
0x000C | UART_LCR (Line Control) |
arch/arm64/src/a527/a527_boot.c
At NuttX Startup: We mount the ROMFS Filesystem (Initial RAM Disk, pic above) containing the NuttX Apps…
How? We safely copy the ROMFS Filesystem from the NuttX Image into the ramdisk
Memory Region. This code comes from NuttX EIC7700X…
// Needed for the `aligned_data` macro
#include <nuttx/compiler.h>
// Just after Bootloader has started NuttX...
void arm64_chip_boot(void) {
// We copy the RAM Disk
qemu_copy_ramdisk();
// Omitted: Other initialisation (MMU, ...)
arm64_mmu_init(true);
...
}
// Copy the RAM Disk from NuttX Image to RAM Disk Region.
static void a527_copy_ramdisk(void) {
const uint8_t aligned_data(8) header[8] = "-rom1fs-";
const uint8_t *limit = (uint8_t *)g_idle_topstack + (256 * 1024);
uint8_t *ramdisk_addr = NULL;
uint8_t *addr;
uint32_t size;
// After Idle Stack Top, search for "-rom1fs-". This is the RAM Disk Address.
// Limit search to 256 KB after Idle Stack Top.
for (addr = g_idle_topstack; addr < limit; addr += 8) {
if (memcmp(addr, header, sizeof(header)) == 0) {
ramdisk_addr = addr;
break;
}
}
// Stop if RAM Disk is missing
if (ramdisk_addr == NULL) {
_err("Missing RAM Disk. Check the initrd padding.");
PANIC();
}
// Read the Filesystem Size from the next 4 bytes (Big Endian)
size = (ramdisk_addr[8] << 24) + (ramdisk_addr[9] << 16) +
(ramdisk_addr[10] << 8) + ramdisk_addr[11] + 0x1f0;
// Filesystem Size must be less than RAM Disk Memory Region
if (size > (size_t)__ramdisk_size) {
_err("RAM Disk Region too small. Increase by %lu bytes.\n", size - (size_t)__ramdisk_size);
PANIC();
}
// Copy the RAM Disk from NuttX Image to RAM Disk Region.
// __ramdisk_start overlaps with ramdisk_addr + size.
a527_copy_overlap(__ramdisk_start, ramdisk_addr, size);
}
// Copy an overlapping memory region. dest overlaps with src + count.
static void a527_copy_overlap(uint8_t *dest, const uint8_t *src, size_t count) {
uint8_t *d = dest + count - 1;
const uint8_t *s = src + count - 1;
if (dest <= src) { _err("dest and src should overlap"); PANIC(); }
while (count--) {
volatile uint8_t c = *s; // Prevent compiler optimization
*d = c;
d--;
s--;
}
}
// RAM Disk Region is defined in Linker Script
extern uint8_t __ramdisk_start[];
extern uint8_t __ramdisk_size[];
Why the aligned addresses?
// Header is aligned to 8 bytes
const uint8_t
aligned_data(8) header[8] =
"-rom1fs-";
// Address is also aligned to 8 bytes
for (
addr = g_idle_topstack;
addr < limit;
addr += 8
) {
// Otherwise this will hit Alignment Fault
memcmp(addr, header, sizeof(header));
...
}
We align our Memory Accesses to 8 Bytes. Otherwise we’ll hit an Alignment Fault…
## Alignment Fault at `memcmp(addr, header, sizeof(header))`
default_fatal_handler:
(IFSC/DFSC) for Data/Instruction aborts:
alignment fault
boards/arm64/a527/avaota-a1/src/a527_bringup.c
At Board Startup: We mount the ROMFS Filesystem (inside RAM) as /dev/ram0…
// At NuttX Startup...
int a527_bringup(void) {
// We Mount the RAM Disk
mount_ramdisk();
...
}
// Mount a RAM Disk defined in ld.script to /dev/ramX. The RAM Disk
// contains a ROMFS filesystem with applications that can be spawned at
// runtime.
static int mount_ramdisk(void) {
struct boardioc_romdisk_s desc;
desc.minor = RAMDISK_DEVICE_MINOR;
desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
desc.sectsize = SECTORSIZE;
desc.image = __ramdisk_start;
int ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);
if (ret < 0) {
syslog(LOG_ERR, "Ramdisk register failed: %s\n", strerror(errno));
syslog(LOG_ERR, "Ramdisk mountpoint /dev/ram%d\n",RAMDISK_DEVICE_MINOR);
syslog(LOG_ERR, "Ramdisk length %lu, origin %lx\n", (ssize_t)__ramdisk_size, (uintptr_t)__ramdisk_start);
}
return ret;
}
// RAM Disk Definition
#define SECTORSIZE 512
#define NSECTORS(b) (((b) + SECTORSIZE - 1) / SECTORSIZE)
#define RAMDISK_DEVICE_MINOR 0
boards/arm64/a527/avaota-a1/scripts/ld.script
We reserve 16 MB of RAM for the ROMFS Filesystem that will host the NuttX Apps…
/* Linker Script: We moved the Paged Pool and added the RAM Disk (16 MB) */
MEMORY {
/* Previously: QEMU boots at 0x4028_0000 */
dram (rwx) : ORIGIN = 0x40800000, LENGTH = 2M
/* Previously: QEMU Paged Memory is at 0x4028_0000 */
/* Why? Because 0x4080_0000 + 2 MB = 0x40A0_0000 */
pgram (rwx) : ORIGIN = 0x40A00000, LENGTH = 4M /* w/ cache */
/* Added the RAM Disk */
ramdisk (rwx) : ORIGIN = 0x40E00000, LENGTH = 16M /* w/ cache */
}
/* We'll reference these in our code */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size = LENGTH(ramdisk);
__ramdisk_end = ORIGIN(ramdisk) + LENGTH(ramdisk);
Also we moved the Paged Pool because the Boot Address has changed to 0x4080_0000.
boards/arm64/a527/avaota-a1/configs/nsh/defconfig
Since we changed the Paged Memory Pool (pgram), we update ARCH_PGPOOL_PBASE and VBASE: configs/knsh/defconfig
## Physical Address of Paged Memory Pool
## Previously: QEMU Paged Memory is at 0x4028_0000
CONFIG_ARCH_PGPOOL_PBASE=0x40A00000
## Virtual Address of Paged Memory Pool
## Previously: QEMU Paged Memory is at 0x4028_0000
CONFIG_ARCH_PGPOOL_VBASE=0x40A00000
NuttX QEMU declares the RAM Size as 128 MB in RAMBANK1_SIZE. We set RAM_SIZE accordingly…
## RAM Size is 128 MB
CONFIG_RAM_SIZE=134217728
Based on the 16550 UART Registers above: We configured the 16550 UART and removed PL011 UART…
CONFIG_16550_ADDRWIDTH=0
CONFIG_16550_REGINCR=4
CONFIG_16550_UART0=y
CONFIG_16550_UART0_BASE=0x02500000
CONFIG_16550_UART0_CLOCK=23040000
CONFIG_16550_UART0_IRQ=125
CONFIG_16550_UART0_SERIAL_CONSOLE=y
CONFIG_16550_UART=y
CONFIG_16550_WAIT_LCR=y
CONFIG_SERIAL_UART_ARCH_MMIO=y
16550_UART0_CLOCK was computed according to these instructions…
NuttX UART Debug Log shows:
dlm = 0x00
dll = 0x0D
We know that:
dlm = 0x00 = (div >> 8)
dll = 0x0D = (div & 0xFF)
Which means:
div = 0x0D
We know that:
baud = 115200
div = (uartclk + (baud << 3)) / (baud << 4)
Therefore:
0x0D = (uartclk + 921600) / 1843200
uartclk = (0x0D * 1843200) - 921600
= 23040000
arch/arm64/src/a527/a527_serial.c
QEMU was using PL011 UART. We switched the Serial Driver to 16550 UART…
// Switch from PL011 UART (QEMU) to 16550 UART
#include <nuttx/serial/uart_16550.h>
// Enable the 16550 Console UART at Startup
void arm64_earlyserialinit(void) {
// Previously for QEMU: pl011_earlyserialinit
u16550_earlyserialinit();
}
// Ditto but not so early
void arm64_serialinit(void) {
// Previous for QEMU: pl011_serialinit
u16550_serialinit();
}