Rust Standard Library on Apache NuttX RTOS

📝 26 Jan 2025

LED Blinky with Rust Standard Library on Apache NuttX RTOS (RustRover IDE)

Freshly Baked: Here’s how we Blink the LED with Rust Standard Library on Apache NuttX RTOS…

// Open the LED Device for NuttX
let fd = open(      // Equivalent to NuttX open()
  "/dev/userleds",  // LED Device
  OFlag::O_WRONLY,  // Write Only
  Mode::empty()     // No Modes
).unwrap();         // Halt on Error

// Define the ioctl() function for Flipping LEDs
const ULEDIOC_SETALL: i32 = 0x1d03;  // ioctl() Command
ioctl_write_int_bad!(  // ioctl() will write One Int Value (LED Bit State)
  led_set_all,         // Name of our New Function
  ULEDIOC_SETALL       // ioctl() Command to send
);

// Flip LED 1 to On
unsafe {             // Be careful of ioctl()
  led_set_all(       // Set the LEDs for...
    fd.as_raw_fd(),  // LED Device
    1                // LED 1 (Bit 0) turns On
  ).unwrap();        // Halt on Error
}  // Equivalent to ioctl(fd, ULEDIOC_SETALL, 1)

// Flip LED 1 to Off: ioctl(fd, ULEDIOC_SETALL, 0)
unsafe { led_set_all(fd.as_raw_fd(), 0).unwrap(); }

Which requires the nix Rust Crate / Library…

## Add the `nix` Rust Crate
## To our NuttX Rust App
$ cd apps/examples/rust/hello
$ cargo add nix --features fs,ioctl

Updating crates.io index
Adding nix v0.29.0 to dependencies
Features: + fs + ioctl

(OK it’s more complicated. Stay tuned)

All this is now possible, thanks to the awesome work by Huang Qi! 🎉

In today’s article, we explain…

§1 Compile our Rust Hello App

How to build NuttX + Rust Standard Library?

Follow the instructions here…

Then run the (thoroughly revamped) Rust Hello App with QEMU RISC-V Emulator…

## Start NuttX on QEMU RISC-V 64-bit
$ qemu-system-riscv64 \
  -semihosting \
  -M virt,aclint=on \
  -cpu rv64 \
  -bios none \
  -kernel nuttx \
  -nographic

## Run the Rust Hello App
NuttShell (NSH) NuttX-12.8.0
nsh> hello_rust_cargo

{"name":"John","age":30}
{"name":"Jane","age":25}
Deserialized: Alice is 28 years old

Pretty JSON:
{
  "name": "Alice",
  "age": 28
}

Hello world from tokio!

Some bits are a little wonky (but will get better)

What’s inside the brand new Rust Hello App? We dive in…

JSON with Serde on Apache NuttX RTOS (Neovim IDE)

§2 JSON with Serde

What’s this Serde?

Think “Serialize-Deserialize”. Serde is a Rust Crate / Library for Serializing and Deserializing our Data Structures. Works with JSON, CBOR, MessagePack, …

This is how we Serialize to JSON in our Hello Rust App: nuttx-apps/lib.rs

// Allow Serde to Serialize and Deserialize a Person Struct
#[derive(Serialize, Deserialize)]
struct Person {
  name: String,  // Containing a Name (string)
  age:  u8,      // And Age (uint8_t)
}  // Note: Rust Strings live in Heap Memory!

// Main Function of our Hello Rust App
#[no_mangle]
pub extern "C" fn hello_rust_cargo_main() {

  // Create a Person Struct
  let john = Person {
    name: "John".to_string(),
    age:  30,
  };

  // Serialize our Person Struct
  let json_str = serde_json // Person Struct
    ::to_string(&john)  // Becomes a String
    .unwrap();          // Halt on Error
  println!("{}", json_str);

Which will print…

NuttShell (NSH) NuttX-12.8.0
nsh> hello_rust_cargo
{"name":"John","age":30}

Now we Deserialize from JSON: lib.rs

// Declare a String with JSON inside
let json_data = r#"
  {
    "name": "Alice",
    "age": 28
  }"#;

// Deserialize our JSON String
// Into a Person Struct
let alice: Person = serde_json // Get Person Struct
  ::from_str(json_data)  // From JSON String
  .unwrap();             // Halt on Error
println!("Deserialized: {} is {} years old",
  alice.name, alice.age);

And we’ll see…

Deserialized: Alice is 28 years old

Serde will also do JSON Formatting: lib.rs

// Serialize our Person Struct
// But neatly please
let pretty_json_str = serde_json // Person Struct
  ::to_string_pretty(&alice)     // Becomes a Formatted String
  .unwrap();                     // Halt on Error
println!("Pretty JSON:\n{}", pretty_json_str);

Looks much neater…

Pretty JSON:
{
  "name": "Alice",
  "age": 28
}

(Serde runs on Rust Core Library, though super messy)

Async Functions with Tokio (Helix Editor + Zellij Workspace)

§3 Async Functions with Tokio

What’s this Tokio? Sounds like a city?

Indeed, “Tokio” is inspired by Tokyo (and Metal I/O)

Tokio … provides a runtime and functions that enable the use of Asynchronous I/O, allowing for Concurrency in regards to Task Completion

Inside our Rust Hello App, here’s how we run Async Functions with Tokio: nuttx-apps/lib.rs

// Use One Single Thread (Current Thread)
// To schedule Async Functions
tokio::runtime::Builder
  ::new_current_thread()  // Current Thread is the Single-Threaded Scheduler
  .enable_all()  // Enable the I/O and Time Functions
  .build()   // Create the Single-Threaded Scheduler
  .unwrap()  // Halt on Error
  .block_on( // Start the Scheduler
    async {  // With this Async Code
      println!("Hello world from tokio!");
  });

// Is it really async? Let's block and find out!
println!("Looping Forever...");
loop {}

We’ll see…

nsh> hello_rust_cargo
Hello world from tokio!
Looping Forever...

(Derived from tokio::main)

Yawn. Tokio looks underwhelming?

Ah we haven’t seen the full power of Tokio Multi-Threaded Async Functions on NuttX…

nsh> hello_rust_cargo
pthread_create
nx_pthread_create

Task 0 sleeping for 1000 ms
Task 1 sleeping for  950 ms
Task 2 sleeping for  900 ms
Task 3 sleeping for  850 ms

Finished time-consuming task
Task 3 stopping
Task 2 stopping
Task 1 stopping
Task 0 stopping

Check this link for the Tokio Async Demo. And it works beautifully on NuttX! (Pic below)

Tokio Async Demo

NuttX has POSIX Threads. Why use Async Functions?

Think Node.js and its Single-Thread Event Loop, making Non-Blocking I/O Calls. Supporting tens of thousands of concurrent connections. (Without costly Thread Context Switching)

Today we can (probably) do the same with NuttX and Async Rust. Assuming POSIX Async I/O works OK with Tokio.

(Tokio calls them “Async Tasks”, sorry we won’t. Because a Task in NuttX means something else)

How will we use Tokio?

Tokio is designed for I/O-Bound Applications where each individual task spends most of its time waiting for I/O.

Which means it’s great for Network Servers. Instead of spawning many POSIX Threads, we spawn a few threads and call Async Functions.

(Check out Tokio Select and Tokio Streams)

LED Blinky with Rust Standard Library on Apache NuttX RTOS (RustRover IDE)

§4 LED Blinky with Nix

We’re running nix on NuttX?

Oh that’s nix Crate that provides Safer Rust Bindings for POSIX / Unix / Linux. (It’s not NixOS)

This is how we add the library to our Rust Hello App…

$ cd ../apps/examples/rust/hello
$ cargo add nix \
  --features fs,ioctl \
  --git https://github.com/lupyuen/nix.git \
  --branch nuttx

Updating git repository `https://github.com/lupyuen/nix.git`
Adding nix (git) to dependencies
Features: + fs + ioctl
34 deactivated features

URL looks sus?

Yep it’s our Bespoke nix Crate. That’s because the Official nix Crate doesn’t support NuttX yet. We made a few tweaks to compile on NuttX. (Explained in the Appendix)

Why call nix?

We’re Blinking the LED on NuttX. We could call the POSIX API direcly from Rust…

let fd = unsafe { libc::open("/dev/userleds", ...) };
unsafe { libc::ioctl(fd, ULEDIOC_SETALL, 1); }
unsafe { libc::close(fd); }

Though it doesn’t look very… Safe. That’s why we call the Safer POSIX Bindings provided by nix. Like so: wip-nuttx-apps/lib.rs

// Open the LED Device for NuttX
let fd = open(      // Equivalent to NuttX open()
  "/dev/userleds",  // LED Device
  OFlag::O_WRONLY,  // Write Only
  Mode::empty()     // No Modes
).unwrap();         // Halt on Error

// Define the ioctl() function for Flipping LEDs
const ULEDIOC_SETALL: i32 = 0x1d03;  // ioctl() Command
ioctl_write_int_bad!(  // ioctl() will write One Int Value (LED Bit State)
  led_set_all,         // Name of our New Function
  ULEDIOC_SETALL       // ioctl() Command to send
);

The code above opens the LED Device, returning an Owned File Descriptor (explained below). It defines a function led_set_all, that will call ioctl() to flip the LED.

Here’s how we call led_set_all to flip the LED: lib.rs

// Flip LED 1 to On
unsafe {             // Be careful of ioctl()
  led_set_all(       // Set the LEDs for...
    fd.as_raw_fd(),  // LED Device
    1                // LED 1 (Bit 0) turns On
  ).unwrap();        // Halt on Error
}  // Equivalent to ioctl(fd, ULEDIOC_SETALL, 1)

We wait Two Seconds, then flip the LED to Off: lib.rs

// Wait 2 seconds
sleep(2);

// Flip LED 1 to Off: ioctl(fd, ULEDIOC_SETALL, 0)
unsafe { led_set_all(fd.as_raw_fd(), 0).unwrap(); }

ULEDIOC_SETALL looks familiar?

We spoke about ULEDIOC_SETALL in an earlier article. And the Rust Code above mirrors the C Version of our Blinky App.

How to run the Rust Blinky App?

  1. Copy the Rust Blinky Files from here…

    lupyuen2/wip-nuttx-apps/examples/rust/hello

    Specifically: Cargo.toml and src/lib.rs

  2. Overwrite our Rust Hello App…

    apps/examples/rust/hello

  3. Rebuild our NuttX Project

    make -j

  4. Then run it with QEMU RISC-V Emulator

    $ qemu-system-riscv64 \
  -semihosting \
  -M virt,aclint=on \
  -cpu rv64 \
  -bios none \
  -kernel nuttx \
  -nographic

NuttShell (NSH) NuttX-12.8.0
nsh> hello_rust_cargo

board_userled: LED 1 set to 1
board_userled: LED 1 set to 0

    NuttX blinks the Emulated LED on QEMU Emulator!

    (See the Complete Log)

How to code Rust Apps for NuttX?

We could open the apps folder in VSCode, but Rust Analyzer won’t work.

Do this instead: VSCode > File > Open Folder > apps/examples/rust/hello. Then Rust Analyzer will work perfectly.

cargo build seems to work, cargo run won’t. Remember to run cargo clippy…

$ cargo clippy
Checking hello v0.1.0 (apps/examples/rust/hello)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.38s

Let’s talk about Owned File Descriptors vs Raw File Descriptors…

Owned File Descriptors vs Raw File Descriptors

§5 Owned File Descriptors

Safety Quiz: Why will this run OK…

// Copied from above: Open the LED Device
let owned_fd =
  open("/dev/userleds", ...)
  .unwrap();  // Returns an Owned File Descriptor

// Copied from above: Set the LEDs via ioctl()
led_set_all(
  owned_fd.as_raw_fd(),  // Extract the Raw File Descriptor
  1                      // Flip LED 1 to On
).unwrap();              // Yep runs OK

But Not This? (Pic above)

// Extract earlier the Raw File Descriptor (from the LED Device)
let raw_fd =
  open("/dev/userleds", ...)  // Open the LED Device
  .unwrap()      // Get the Owned File Descriptor
  .as_raw_fd();  // Which becomes a Raw File Descriptor

// Set the LEDs via ioctl()
led_set_all(
  raw_fd,    // Use the earlier Raw File Descriptor
  1          // Flip LED 1 to On
).unwrap();  // Oops will fail!

The Second Snippet will fail with EBADF Error…

nsh> hello_rust_cargo
thread '<unnamed>' panicked at src/lib.rs:32:33:
called `Result::unwrap()` on an `Err` value: EBADF
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

There’s something odd about Raw File Descriptors vs Owned File Descriptors… Fetching the Raw One too early might cause EBADF Errors. Here’s why…

What’s a Raw File Descriptor?

In NuttX and POSIX: Raw File Descriptor is a Plain Integer that specifies an I/O Stream…

File DescriptorI/O Stream
0Standard Input
1Standard Output
2Standard Error
3/dev/userleds
(assuming we opened it)

What about Owned File Descriptor?

In Rust: Owned File Descriptor is a Rust Object, wrapped around a Raw File Descriptor.

And Rust Objects shall be Automatically Dropped, when they go out of scope. (Unlike Integers)

Causing the Second Snippet to fail?

Exactly! open() returns an Owned File Descriptor…

// Owned File Descriptor becomes Raw File Descriptor
let raw_fd =
  open("/dev/userleds", ...)  // Open the LED Device
  .unwrap()      // Get the Owned File Descriptor
  .as_raw_fd();  // Which becomes a Raw File Descriptor

And we turned it into Raw File Descriptor. (The Plain Integer, not the Rust Object)

Oops! Our Owned File Descriptor goes Out Of Scope and gets dropped by Rust…

Our Owned File Descriptor goes Out Of Scope and gets dropped by Rust

Thus Rust will helpfully close /dev/userleds. Since it’s closed, our Raw File Descriptor becomes invalid…

// Set the LEDs via ioctl()
led_set_all(
  raw_fd,    // Use the (closed) Raw File Descriptor
  1          // Flip LED 1 to On
).unwrap();  // Oops will fail with EBADF Error!

Resulting in the EBADF Error. ioctl() failed because /dev/userleds is already closed!

Lesson Learnt: Be careful with Owned File Descriptors. They’re super helpful for Auto-Closing our files. But might have strange consequences.

Rustix is another popular POSIX Wrapper. We take a peek…

Nix vs Rustix

§6 Nix vs Rustix

Is there a Safer Way to call ioctl()?

Calling ioctl() from Rust will surely get messy: It’s an Unsafe Call that might cause bad writes into the NuttX Kernel! (If we’re not careful)

At the top of the article, we saw nix crate calling ioctl(). Now we look at Rustix calling ioctl(): rustix/fs/ioctl.rs

// Let's implement ioctl(fd, BLKSSZGET, &output)
// In Rustix: ioctl() is also unsafe
unsafe {
  // Create an "Ioctl Getter"
  // That will read data thru ioctl()
  let ctl = ioctl::Getter::<  // Ioctl Getter has 2 attributes...
    ioctl::BadOpcode<  // Attribute #1: Ioctl Command Code
      { c::BLKSSZGET } // Which is "Fetch the Logical Block Size of a Block Device"
    >,
    c::c_uint  // Attribute #2: Ioctl Getter will read a uint32_t thru ioctl()
  >::new();    // Create the Ioctl Getter

  // Now that we have the Ioctl Getter
  // We call ioctl() on the File Descriptor
  // Equivalent to: ioctl(fd, BLKSSZGET, &output) ; return output
  ioctl::ioctl(
    fd,  // Borrowed File Descriptor (safer than Raw)
    ctl  // Ioctl Getter
  ) // Returns the Value Read (Or Error)
}

(Based on Rustix Docs)

(Rustix Ioctl passes a Borrowed File Descriptor, safer than Raw)

Nix vs Rustix: They feel quite similar?

Actually Nix was previously a lot simpler, supporting only Raw File Descriptors. (Instead of Owned File Descriptors)

Today, Nix is moving to Owned File Descriptors due to I/O Safety. Bummer it means Nix is becoming more Rustix-like…

What’s our preference: Nix or Rustix?

Hmmm we’re still pondering. Rustix is newer (pic above), but it’s also more complex (based on Lines of Code). It might hinder our porting to NuttX.

Which would you choose? Lemme know! 🙏

Nix vs Rustix: Lines of Code

(Rustix on NuttX: Will it run? Nope not yet)

(no1wudi/nuttx-rs shows potential)

(Rust Embedded HAL might be a bad fit)

§7 What’s Next

Upcoming: Slint Rust GUI for NuttX 🎉

Upcoming: Slint Rust GUI for NuttX 🎉

What platforms are supported for NuttX + Rust Standard Library? How about SBCs?

Arm and RISC-V (32-bit and 64-bit). Check this doc for updates.

Sorry 64-bit RISC-V Kernel Build is not supported yet. So it won’t run on RISC-V SBCs like Ox64 BL808 and Oz64 SG2000.

Sounds like we need plenty of Rust Testing? For every NuttX Platform?

Yeah maybe we need Daily Automated Testing of NuttX + Rust Standard Library on NuttX Build Farm?

With QEMU Emulator or a Real Device?

And when the Daily Test fails: How to Auto-Rewind the Build and discover the Breaking Commit? Hmmm…

Many Thanks to the awesome NuttX Admins and NuttX Devs! And My Sponsors, for sticking with me all these years.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.org/src/rust7.md

NuttX with Rust Standard Library

§8 Appendix: Build NuttX for Rust Standard Library

Follow these steps to build NuttX bundled with Rust Standard Library…

(Based on the Official Doc)

(Remember to install RISC-V Toolchain and RISC-V QEMU)

## Install Rust: https://rustup.rs/
## Select "Standard Installation"
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
. "$HOME/.cargo/env"

## Switch to the Nightly Toolchain
rustup update
rustup toolchain install nightly
rustup default nightly

## Should show `rustc 1.86.0-nightly` or later
rustc --version

## Install the Nightly Toolchain
rustup component add rust-src --toolchain nightly-x86_64-unknown-linux-gnu
## For macOS: rustup component add rust-src --toolchain nightly-aarch64-apple-darwin

## Download the NuttX Kernel and Apps
git clone https://github.com/apache/nuttx
git clone https://github.com/apache/nuttx-apps apps
cd nuttx

## Configure NuttX for RISC-V 64-bit QEMU with LEDs
## (Alternatively: rv-virt:nsh64 or rv-virt:nsh or rv-virt:leds)
tools/configure.sh rv-virt:leds64

## Disable Floating Point: CONFIG_ARCH_FPU
kconfig-tweak --disable CONFIG_ARCH_FPU

## Enable CONFIG_SYSTEM_TIME64 / CONFIG_FS_LARGEFILE / CONFIG_DEV_URANDOM / CONFIG_TLS_NELEM = 16
kconfig-tweak --enable CONFIG_SYSTEM_TIME64
kconfig-tweak --enable CONFIG_FS_LARGEFILE
kconfig-tweak --enable CONFIG_DEV_URANDOM
kconfig-tweak --set-val CONFIG_TLS_NELEM 16

## Enable the Hello Rust Cargo App
## Increase the App Stack Size from 2 KB to 16 KB (especially for 64-bit platforms)
kconfig-tweak --enable CONFIG_EXAMPLES_HELLO_RUST_CARGO
kconfig-tweak --set-val CONFIG_EXAMPLES_HELLO_RUST_CARGO_STACKSIZE 16384

## Update the Kconfig Dependencies
make olddefconfig

## Build NuttX
make -j

## If it fails with "Mismatched Types":
## Patch the file `fs.rs` (see below)

## Start NuttX on QEMU RISC-V 64-bit
qemu-system-riscv64 \
  -semihosting \
  -M virt,aclint=on \
  -cpu rv64 \
  -bios none \
  -kernel nuttx \
  -nographic

## Inside QEMU: Run our Hello Rust App
hello_rust_cargo

We’ll see this in QEMU RISC-V Emulator…

NuttShell (NSH) NuttX-12.8.0
nsh> hello_rust_cargo

{"name":"John","age":30}
{"name":"Jane","age":25}
Deserialized: Alice is 28 years old
Pretty JSON:
{
  "name": "Alice",
  "age": 28
}
Hello world from tokio!

To Quit QEMU: Press Ctrl-a then x

(See the Ubuntu Build Log)

(See the macOS Build Log)

(Also works for 32-bit rv-virt:leds)

Troubleshooting The Rust Build

How did we port Rust Standard Library to NuttX? Details here…

Tokio Async Threading

§9 Appendix: Tokio Async Threading

Earlier we saw Tokio’s Single-Threaded Scheduler, running on the Current Thread…

// Use One Single Thread (Current Thread)
// To schedule Async Functions
tokio::runtime::Builder
  ::new_current_thread()  // Current Thread is the Single-Threaded Scheduler
  .enable_all()  // Enable the I/O and Time Functions
  .build()   // Create the Single-Threaded Scheduler
  .unwrap()  // Halt on Error
  .block_on( // Start the Scheduler
    async {  // With this Async Code
      println!("Hello world from tokio!");
  });

// Is it really async? Let's block and find out!
println!("Looping Forever...");
loop {}

And it ain’t terribly exciting…

nsh> hello_rust_cargo
Hello world from tokio!
Looping Forever...

Now we try Tokio’s Multi-Threaded Scheduler. And we create One New POSIX Thread for the Scheduler: wip-nuttx-apps/lib.rs

// Run 4 Async Functions in the Background
// By creating One New POSIX Thread
// Based on https://tokio.rs/tokio/topics/bridging
fn test_async() {

  // Create a Multi-Threaded Scheduler
  // Containing One New POSIX Thread
  let runtime = tokio::runtime::Builder
    ::new_multi_thread() // Multi-Threaded Scheduler
    .worker_threads(1)   // With One New POSIX Thread for our Scheduler
    .enable_all() // Enable the I/O and Time Functions
    .build()      // Create the Multi-Threaded Scheduler
    .unwrap();    // Halt on Error

  // Create 4 Async Functions
  // Remember their Async Handles
  let mut handles = Vec::with_capacity(4);
  for i in 0..4 {
    handles.push(        // Remember the Async Handles
      runtime.spawn(     // Start in the Background
        my_bg_task(i))); // Our Async Function
  }

  // Pretend to be busy while Async Functions execute (in the background)
  // We wait 750 milliseconds
  std::thread::sleep(
    tokio::time::Duration::from_millis(750));
  println!("Finished time-consuming task.");

  // Wait for All Async Functions to complete
  for handle in handles {
    runtime
      .block_on(handle)  // Wait for One Async Function to complete
      .unwrap();
  }
}

// Our Async Function that runs in the background...
// If i=0: Sleep for 1000 ms
// If i=1: Sleep for  950 ms
// If i=2: Sleep for  900 ms
// If i=3: Sleep for  850 ms
async fn my_bg_task(i: u64) {
  let millis = 1000 - 50 * i;
  println!("Task {} sleeping for {} ms.", i, millis);
  tokio::time::sleep(
    tokio::time::Duration::from_millis(millis)
  ).await;  // Wait for sleep to complete
  println!("Task {} stopping.", i);
}

// Needed by Tokio Multi-Threaded Scheduler
#[no_mangle]
pub extern "C" fn pthread_set_name_np() {}

How to run the Tokio Demo?

  1. Copy the Tokio Demo Files from here…

    lupyuen2/wip-nuttx-apps/examples/rust/hello

    Specifically: Cargo.toml and src/lib.rs

  2. Overwrite our Rust Hello App…

    apps/examples/rust/hello

  3. Rebuild our NuttX Project

    make -j

  4. Then run it with QEMU RISC-V Emulator

    $ qemu-system-riscv64 \
  -semihosting \
  -M virt,aclint=on \
  -cpu rv64 \
  -bios none \
  -kernel nuttx \
  -nographic

NuttShell (NSH) NuttX-12.8.0
nsh> hello_rust_cargo

  5. We’ll see Four Async Functions, running on One New POSIX Thread…

    nsh> hello_rust_cargo
pthread_create
nx_pthread_create

Task 0 sleeping for 1000 ms
Task 1 sleeping for  950 ms
Task 2 sleeping for  900 ms
Task 3 sleeping for  850 ms

Finished time-consuming task
Task 3 stopping
Task 2 stopping
Task 1 stopping
Task 0 stopping

    (See the Complete Log)

    (Explained in Tokio Docs)

  6. See the call to pthread_create, which calls nx_pthread_create? It means that Tokio is actually calling NuttX to create One POSIX Thread! (For the Multi-Threaded Scheduler)

  7. Yep it’s consistent with our Reverse Engineering of Tokio…

    “Snooping Tokio on NuttX”

What if we increase the Worker Threads? From 1 to 2?

// Two Worker Threads instead of One
let runtime = tokio::runtime::Builder
  ::new_multi_thread() // New Multi-Threaded Scheduler
  .worker_threads(2)   // With Two New POSIX Threads for our Scheduler

The output looks exactly the same…

nsh> hello_rust_cargo
pthread_create
nx_pthread_create
pthread_create
nx_pthread_create

Task 0 sleeping for 1000 ms
Task 1 sleeping for  950 ms
Task 2 sleeping for  900 ms
Task 3 sleeping for  850 ms

Finished time-consuming task
Task 3 stopping
Task 2 stopping
Task 1 stopping
Task 0 stopping

Except that we see Two Calls to pthread_create and nx_pthread_create. Tokio called NuttX to create Two POSIX Threads. (For the Multi-Threaded Scheduler)

How did we log pthread_create?

Inside NuttX Kernel: We added Debug Code to pthread_create and nx_pthread_create

// At https://github.com/apache/nuttx/blob/master/libs/libc/pthread/pthread_create.c#L88
#include <debug.h>
int pthread_create(...) {
  _info("pthread_entry=%p, arg=%p", pthread_entry, arg);

// At https://github.com/apache/nuttx/blob/master/sched/pthread/pthread_create.c#L179
#include <debug.h>
int nx_pthread_create(...) {
  _info("entry=%p, arg=%p", entry, arg);

LED Blinky with Rust Standard Library on Apache NuttX RTOS (RustRover IDE)

§10 Appendix: Porting Nix to NuttX

What happens when we call nix crate as-is on NuttX?

Earlier we said that we Customised the nix Crate to run on NuttX.

Why? Let’s build our Rust Blinky App with the Original nix Crate…

$ pushd ../apps/examples/rust/hello
$ cargo add nix --features fs,ioctl

Adding nix v0.29.0 to dependencies
Features: + fs + ioctl
33 deactivated features

$ popd
$ make -j

error[E0432]: unresolved import `self::const`
  -->   errno.rs:19:15
19 | pub use self::consts::*;
   |               ^^^^^^ could not find `consts` in `self`

error[E0432]: unresolved import `self::Errno`
   -->  errno.rs:198:15
198 |     use self::Errno::*;
    |               ^^^^^ could not find `Errno` in `self`

error[E0432]: unresolved import `crate::errno::Errno`
 -->  fcntl.rs:2:5
2 | use crate::errno::Errno;
  |     ^^^^^^^^^^^^^^-----
  |     no `Errno` in `errno`

Plus many errors. That’s why we Customised the nix Crate for NuttX…

$ cd ../apps/examples/rust/hello
$ cargo add nix \
  --features fs,ioctl \
  --git https://github.com/lupyuen/nix.git \
  --branch nuttx

Updating git repository `https://github.com/lupyuen/nix.git`
Adding nix (git) to dependencies
Features: + fs + ioctl
34 deactivated features

Here’s how…

  1. For Easier Porting: We cloned nix locally…

    git clone \
  https://github.com/lupyuen/nix \
  --branch nuttx
cd ../apps/examples/rust/hello
cargo add nix \
  --features fs,ioctl \
  --path $HOME/nix

  2. We extended errno.rs, copying the FreeBSD Section [cfg(target_os = “freebsd”)] to NuttX Section [cfg(target_os = “nuttx”)].

    (We removed the bits that don’t exist on NuttX)

  3. NuttX seems to have a similar POSIX Profile to Redox OS? We changed plenty of code to look like this: sys/time.rs

    // NuttX works like Redox OS
#[cfg(not(any(target_os = "redox",
              target_os = "nuttx")))]
pub const UTIME_OMIT: TimeSpec = ...

  4. For NuttX ioctl(): It works more like BSD (second parameter is int) than Linux (second parameter is long): sys/ioctl/mod.rs

    // NuttX ioctl() works like BSD
#[cfg(any(bsd,
          solarish,
          target_os = "haiku",
          target_os = "nuttx"))]
#[macro_use]
mod bsd;

// Nope, NuttX ioctl() does NOT work like Linux
#[cfg(any(linux_android,
          target_os = "fuchsia",
          target_os = "redox"))]
#[macro_use]
mod linux;

  5. Here are the files we modified for NuttX…

    (Supporting fs and ioctl features only)

    All Modified Files

    errno.rs

    fcntl.rs

    unistd.rs

    sys/stat.rs

    sys/statvfs.rs

    sys/mod.rs

    sys/time.rs

    sys/ioctl/bsd.rs

    sys/ioctl/mod.rs

Troubleshooting nix ioctl() on NuttX

To figure out if nix passes ioctl() parameters correctly to NuttX: We insert Ioctl Debug Code into NuttX Kernel…

// At https://github.com/apache/nuttx/blob/master/fs/vfs/fs_ioctl.c#L261
#include <debug.h>
int ioctl(int fd, int req, ...) {
  _info("fd=0x%x, req=0x%x", fd, req);

Which Ioctl Macro shall we call in nix? We tried ioctl_none…

const ULEDIOC_SETALL: i32 = 0x1d03;
ioctl_none!(led_on, ULEDIOC_SETALL, 1);
unsafe { led_on(fd).unwrap(); }

But the Ioctl Command Code got mangled up (0x201d0301 should be 0x1d03)

NuttShell (NSH) NuttX-12.8.0
nsh> hello_rust_cargo
fd=3
ioctl: fd=0x3, req=0x201d0301

thread '<unnamed>' panicked at src/lib.rs:31:25:
called `Result::unwrap()` on an `Err` value: ENOTTY
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Then we tried ioctl_write_int…

const ULEDIOC_SETALL: i32 = 0x1d03;
ioctl_write_int!(led_on, ULEDIOC_SETALL, 1);
unsafe { led_on(fd, 1).unwrap(); }

Nope the Ioctl Command Code is still mangled (0x801d0301 should be 0x1d03)

nsh> hello_rust_cargo
ioctl: fd=0x3, req=0x801d0301
thread '<unnamed>' panicked at src/lib.rs:30:28:
called `Result::unwrap()` on an `Err` value: ENOTTY

Finally this works: ioctl_write_int_bad…

const ULEDIOC_SETALL: i32 = 0x1d03;
ioctl_write_int_bad!(led_set_all, ULEDIOC_SETALL);

// Equivalent to ioctl(fd, ULEDIOC_SETALL, 1)
unsafe { led_set_all(fd, 1).unwrap(); }

// Equivalent to ioctl(fd, ULEDIOC_SETALL, 0)
unsafe { led_set_all(fd, 0).unwrap(); }

Ioctl Command Code 0x1d03 is hunky dory yay!

NuttShell (NSH) NuttX-12.8.0
nsh> hello_rust_cargo
fd=3
ioctl: fd=0x3, req=0x1d03
board_userled: LED 1 set to 1
board_userled: LED 2 set to 0
board_userled: LED 3 set to 0

ioctl: fd=0x3, req=0x1d03
board_userled: LED 1 set to 0
board_userled: LED 2 set to 0
board_userled: LED 3 set to 0

Nix vs Rustix

§11 Appendix: Porting Rustix to NuttX

Will Rustix run on NuttX?

Nope not yet…

$ cd ../apps/examples/rust/hello
$ cargo add rustix \
  --features fs \
  --git https://github.com/lupyuen/rustix.git \
  --branch nuttx

Updating git repository `https://github.com/lupyuen/rustix.git`
Adding rustix (git) to dependencies
Features: + alloc + fs + std + use-libc-auxv
29 deactivated features

We tried compiling this code…

#[no_mangle]
pub extern "C" fn hello_rust_cargo_main() {
  use rustix::fs::{Mode, OFlags};
  let file = rustix::fs::open(
    "/dev/userleds",
    OFlags::WRONLY,
    Mode::empty(),
  )
  .unwrap();
  println!("file={file:?}");
}

But it fails…

error[E0432]: unresolved import `libc::strerror_r`
  --> .cargo/registry/src/index.crates.io-1949cf8c6b5b557f/errno-0.3.10/src/unix.rs:16:33
   |
16 | use libc::{self, c_int, size_t, strerror_r, strlen};
   |                                 ^^^^^^^^^^
   |                                 |
   |                                 no `strerror_r` in the root
   |                                 help: a similar name exists in the module: `strerror`

Seems we need to fix libc::strerror_r for NuttX? Or maybe the errno crate.

Async Functions with Tokio (Helix Editor + Zellij Workspace)

§12 Appendix: Snooping Tokio on NuttX

In this section, we discover how Tokio works under the hood. Does it really call POSIX Functions in NuttX?

First we obtain the RISC-V Disassembly of our NuttX Image, bundled with the Hello Rust App. We trace the NuttX Build: make V=1

make distclean
tools/configure.sh rv-virt:leds64

## Disable CONFIG_ARCH_FPU
kconfig-tweak --disable CONFIG_ARCH_FPU

## Enable CONFIG_SYSTEM_TIME64 / CONFIG_FS_LARGEFILE / CONFIG_DEV_URANDOM / CONFIG_TLS_NELEM = 16
kconfig-tweak --enable CONFIG_SYSTEM_TIME64
kconfig-tweak --enable CONFIG_FS_LARGEFILE
kconfig-tweak --enable CONFIG_DEV_URANDOM
kconfig-tweak --set-val CONFIG_TLS_NELEM 16

## Enable Hello Rust Cargo App, increase the Stack Size
kconfig-tweak --enable CONFIG_EXAMPLES_HELLO_RUST_CARGO
kconfig-tweak --set-val CONFIG_EXAMPLES_HELLO_RUST_CARGO_STACKSIZE 16384

## Update the Kconfig Dependencies
make olddefconfig

## Build NuttX with Tracing Enabled
make V=1

(See the Build Log)

According to the Make Trace: NuttX Build does this…

## Discard the Rust Debug Symbols
cd apps/examples/rust/hello
cargo build \
  --release \
  -Zbuild-std=std,panic_abort \
  --manifest-path apps/examples/rust/hello/Cargo.toml \
  --target riscv64imac-unknown-nuttx-elf

## Generate the Linker Script
riscv-none-elf-gcc \
  -E \
  -P \
  -x c \
  -isystem nuttx/include \
  -D__NuttX__ \
  -DNDEBUG \
  -D__KERNEL__  \
  -I nuttx/arch/risc-v/src/chip \
  -I nuttx/arch/risc-v/src/common \
  -I nuttx/sched \
  nuttx/boards/risc-v/qemu-rv/rv-virt/scripts/ld.script \
  -o  nuttx/boards/risc-v/qemu-rv/rv-virt/scripts/ld.script.tmp

## Link Rust App into NuttX
riscv-none-elf-ld \
  --entry=__start \
  -melf64lriscv \
  --gc-sections \
  -nostdlib \
  --cref \
  -Map=nuttx/nuttx.map \
  --print-memory-usage \
  -Tnuttx/boards/risc-v/qemu-rv/rv-virt/scripts/ld.script.tmp  \
  -L nuttx/staging \
  -L nuttx/arch/risc-v/src/board  \
  -o nuttx/nuttx   \
  --start-group \
  -lsched \
  -ldrivers \
  -lboards \
  -lc \
  -lmm \
  -larch \
  -lm \
  -lapps \
  -lfs \
  -lbinfmt \
  -lboard xpack-riscv-none-elf-gcc-13.2.0-2/lib/gcc/riscv-none-elf/13.2.0/rv64imac/lp64/libgcc.a apps/examples/rust/hello/target/riscv64imac-unknown-nuttx-elf/release/libhello.a \
  --end-group

Ah NuttX Build calls cargo build --release, stripping the Debug Symbols. We change it to cargo build and dump the RISC-V Disassembly…

## Preserve the Rust Debug Symbols
pushd ../apps/examples/rust/hello
cargo build \
  -Zbuild-std=std,panic_abort \
  --manifest-path apps/examples/rust/hello/Cargo.toml \
  --target riscv64imac-unknown-nuttx-elf
popd

## Generate the Linker Script
riscv-none-elf-gcc \
  -E \
  -P \
  -x c \
  -isystem nuttx/include \
  -D__NuttX__ \
  -DNDEBUG \
  -D__KERNEL__  \
  -I nuttx/arch/risc-v/src/chip \
  -I nuttx/arch/risc-v/src/common \
  -I nuttx/sched \
  nuttx/boards/risc-v/qemu-rv/rv-virt/scripts/ld.script \
  -o  nuttx/boards/risc-v/qemu-rv/rv-virt/scripts/ld.script.tmp

## Link Rust App into NuttX
riscv-none-elf-ld \
  --entry=__start \
  -melf64lriscv \
  --gc-sections \
  -nostdlib \
  --cref \
  -Map=nuttx/nuttx.map \
  --print-memory-usage \
  -Tnuttx/boards/risc-v/qemu-rv/rv-virt/scripts/ld.script.tmp  \
  -L nuttx/staging \
  -L nuttx/arch/risc-v/src/board  \
  -o nuttx/nuttx   \
  --start-group \
  -lsched \
  -ldrivers \
  -lboards \
  -lc \
  -lmm \
  -larch \
  -lm \
  -lapps \
  -lfs \
  -lbinfmt \
  -lboard xpack-riscv-none-elf-gcc-13.2.0-2/lib/gcc/riscv-none-elf/13.2.0/rv64imac/lp64/libgcc.a apps/examples/rust/hello/target/riscv64imac-unknown-nuttx-elf/debug/libhello.a \
  --end-group

## Dump the disassembly to nuttx.S
riscv-none-elf-objdump \
  --syms --source --reloc --demangle --line-numbers --wide \
  --debugging \
  nuttx \
  >leds64-debug-nuttx.S \
  2>&1

(See the Build Log)

Which produces the Complete NuttX Disassembly: leds64-debug-nuttx.S

Whoa the Complete NuttX Disassembly is too huge to inspect!

Let’s dump the RISC-V Disassembly of the Rust Part only: libhello.a

## Dump the libhello.a disassembly to libhello.S
riscv-none-elf-objdump \
  --syms --source --reloc --demangle --line-numbers --wide \
  --debugging \
  apps/examples/rust/hello/target/riscv64imac-unknown-nuttx-elf/debug/libhello.a \
  >libhello.S \
  2>&1

Which produces the (much smaller) Rust Disassembly: libhello.S

Is Tokio calling NuttX to create POSIX Threads? We search libhello.S for pthread_create…

.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/pal/unix/thread.rs:85

let ret = libc::pthread_create(&mut native, &attr, thread_start, p as *mut _);

auipc a0, 0x0 122: R_RISCV_PCREL_HI20 std::sys::pal::unix::thread::Thread::new::thread_start
mv    a2, a0 126: R_RISCV_PCREL_LO12_I .Lpcrel_hi254
add   a0, sp, 132
add   a1, sp, 136
sd    a1, 48(sp)
auipc ra, 0x0 130: R_RISCV_CALL_PLT pthread_create

OK that’s the Rust Standard Library calling pthread_create to create a new Rust Thread.

How are Rust Threads created in Rust Standard Library? Like this: std/thread/mod.rs

// spawn_unchecked_ creates a new Rust Thread
unsafe fn spawn_unchecked_<'scope, F, T>(
  let my_thread = Thread::new(id, name);

And spawn_unchecked is called by Tokio, according to our Rust Disassembly…

<core::ptr::drop_in_place<std::thread::Builder::spawn_unchecked_::MaybeDangling<tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}>>>:

.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:523

add   sp, sp, -16
sd    ra, 8(sp)
sd    a0, 0(sp)
auipc ra, 0x0 6: R_RISCV_CALL_PLT <std::thread::Builder::spawn_unchecked_::MaybeDangling<T> as core::ops::drop::Drop>::drop

Yep it checks out: Tokio calls Rust Standard Library, which calls NuttX to create POSIX Threads!

Are we sure that Tokio creates a POSIX Thread? Not a NuttX Task?

We run hello_rust_cargo & to put it in the background…

nsh> hello_rust_cargo &
Hello world from tokio!

nsh> ps
  PID GROUP PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK            STACK    USED FILLED COMMAND
    0     0   0 FIFO     Kthread   - Ready              0000000000000000 0001904 0000712  37.3%  Idle_Task
    2     2 100 RR       Task      - Running            0000000000000000 0002888 0002472  85.5%! nsh_main
    4     4 100 RR       Task      - Ready              0000000000000000 0007992 0006904  86.3%! hello_rust_cargo

ps says that there’s only One Single NuttX Task hello_rust_cargo. And no other NuttX Tasks.

(See the Complete Log)