Forgejo Git Forge for Apache NuttX RTOS (Experimental)

📝 12 Jan 2025

Life Without GitHub: What’s it like? Today we talk about Forgejo Git Forge, and whether Apache NuttX RTOS could possibly switch from GitHub to a self-hosted Git Forge.

What’s this Forgejo? Why is it a “Git Forge”?

Think GitHub… But Open-Source and Self-Hosted! (GoLang Web + PostgreSQL Database)

Forgejo is a Git Forge, the server code that will publicly host and share a Git Repo. (Including our NuttX Repo)

Why explore Forgejo for NuttX?

• If GitHub Breaks: What’s our Contingency Plan?

• GitHub is Blocked in some parts of the world…

• Can we Mirror our NuttX Repo outside GitHub? So NuttX Community becomes more inclusive?

• Also: We’re outgrowing our Budget Limits at GitHub, might need to move out

  (Though ASF Policy might not allow it)

§1 NuttX On Forgejo

Installing our own Git Forge: Is it easy?

Yep! Here’s our experiment of NuttX on Forgejo (pic below)

https://nuttx-forge.org

Bringing up our Forgejo Server was plain-sailing (especially on Docker)

## Download the Forgejo Docker Image
## And our Docker Compose Config
docker pull codeberg.org/forgejo/forgejo:9
git clone https://nuttx-forge.org/lupyuen/nuttx-forgejo

## Up the Forgejo Server
cd nuttx-forgejo
sudo docker compose up

## To Configure: Browse to http://localhost:3002

• “Install our Forgejo Server”

• Based on the excellent Forgejo Docs

Our Git Forge is running on Plain Old SQLite database. Later we might Upgrade to PostgreSQL.

§2 Works The Same

Is it easy to use our own Git Forge?

Yes Forgejo is pleasantly Gittish-Hubbish. Inside Forgejo: Pull Requests and Issues look familiar…

(More on Pull Requests and Issues)

Forgejo is fully compatible with our Git Command-Line Tools (and VSCode)

## Download the NuttX Mirror Repo
## From our Forgejo Server
git clone \
  https://nuttx-forge.org/nuttx/nuttx-mirror

## Also works for SSH (instead of HTTPS)
## But SSH isn't enabled on our server yet
git clone \
  [email protected]:nuttx/nuttx-mirror

Have we seen this somewhere?

• Codeberg is powered by Forgejo

• Fedora Linux Project is moving to Forgejo

§3 Coexist With GitHub

Will our Git Forge coexist with GitHub?

Ah now it gets tricky. Ideally we should allow GitHub to coexist with our Git Forge, synced both ways (pic above)

• GitHub → Our Git Forge

  NuttX Repo at GitHub shall sync down regularly to Our Git Forge

  (So NuttX Devs can pull updates if GitHub breaks)

• Our Git Forge → GitHub

  Pull Requests at our Git Forge shall be pushed up to NuttX Repo at GitHub

  (Allow Local Changes in our Git Forge to be synced back)

Forgejo works great for syncing NuttX Repo from GitHub. We configured Forgejo to Auto-Sync from GitHub every hour…

Oops Forgejo creates a Read-Only Mirror. That won’t allow Pull Requests!

Thus we create our own Read-Write Mirror of NuttX Repo…

Forgejo won’t auto-sync our Read-Write Repo. We wrote our own Syncing Script, that works without GitHub.

(More about the Sync Script in a while)

But Pull Requests shall be synced back to GitHub?

Indeed, we’ll probably call GitHub API to send the Pull Requests back to GitHub. (Pic above)

With this setup, we can’t allow Pull Requests to be locally merged at Forgejo. Instead, Pull Requests shall be merged at GitHub. Then Forgejo shall auto-sync the updates into our Git Forge.

That’s why we need Two Mirror Repos: Read-Only and Read-Write…

(Blocked by Corporate Firewall? Git Mirroring might help)

§4 Continuous Integration

Will our Git Forge run CI Checks on Pull Requests?

GitHub Actions CI (Continuous Integration) becomes a Sticky Issue with Forgejo…

• Forgejo will import GitHub Actions Workflows and execute them (pic above)

• But we don’t have a CI Server to execute the CI Workflow (yet)

• Some GitHub Workflows are Not Supported (NuttX Build Rules arch.yml)

Any special requirement for CI Server?

Our CI Server needs to be Ubuntu x64, hardened for security.

Why? During PR Submission: Our CI Workflow might execute the scripts and code submitted by NuttX Devs.

If we don’t secure our CI Server, we might create Security Problems in our server.

Securing our CI Server is probably the toughest part of our Git Forge Migration. That’s why GitHub is expensive!

(Shall we move NuttX Scripts to a More Secure Repo?)

§5 Sync our Read-Write Mirror

Forgejo won’t Auto-Sync our Read-Write Mirror. How to sync it?

We run a script to Sync the Git Commits…

• From our Read-Only Mirror nuttx-mirror

• To the Read-Write Mirror nuttx-update

• So it will work even when GitHub breaks

• Commit History shall be 100% identical to GitHub

  (Including the Commit Hashes)

## Sync Once: From Read-Only Mirror to Read-Write Mirror
git clone https://nuttx-forge.org/lupyuen/nuttx-forgejo
cd nuttx-forgejo
./sync-mirror-to-update.sh

## Or to Sync Periodically
## ./run.sh

(See the Sync Log)

Our script begins like this: sync-mirror-to-update.sh

## Sync the Git Commits
## From our Read-Only Mirror Repo (nuttx-mirror)
## To the Read-Write Mirror Repo (nuttx-update)
set -e  ## Exit when any command fails

## Checkout the Upstream and Downstream Repos
## Upstream is Read-Only Mirror
## Downstream is Read-Write Mirror
tmp_dir=/tmp/sync-mirror-to-update
rm -rf $tmp_dir ; mkdir $tmp_dir ; cd $tmp_dir
git clone \
  git@nuttx-forge:nuttx/nuttx-mirror \
  upstream
git clone \
  git@nuttx-forge:nuttx/nuttx-update \
  downstream

## Find the First Commit to Sync
pushd upstream
upstream_commit=$(git rev-parse HEAD)
git --no-pager log -1
popd
pushd downstream
downstream_commit=$(git rev-parse HEAD)
git --no-pager log -1
popd

## If no new Commits to Sync: Quit
if [[ "$downstream_commit" == "$upstream_commit" ]]; then
  echo "No New Commits to Sync" ; exit
fi

## Up Next: Sync Downstream Repo with Upstream

How to Sync the Two Repos? Git Pull will do! sync-mirror-to-update.sh

## Apply the Upstream Commits to Downstream Repo
pushd downstream
git pull \
  git@nuttx-forge:nuttx/nuttx-mirror \
  master
git status
popd

## Commit the Downstream Repo
pushd downstream
git push -f
popd

## Verify that Upstream and Downstream Commits are identical
echo "Updated Downstream Commit"
pushd downstream
git pull
downstream_commit2=$(git rev-parse HEAD)
git --no-pager log -1
popd

## If Not Identical: We have a problem, need to Hard-Revert the Commits
if [[ "$downstream_commit2" != "$upstream_commit" ]]; then
  echo "Sync Failed: Upstream and Downstream Commits don't match!" ; exit 1
fi

(See the Sync Log)

What if we accidentally Merge a PR downstream? And our Read-Write Mirror goes out of sync?

If our Read-Write Mirror goes out of sync: We Hard-Revert the Commits in our Read-Write Mirror. Which will keep it in sync again with the Read-Only Mirror…

## Download our Read-Write Mirror
## Which contains Conflicting Commits
$ git clone git@nuttx-forge:nuttx/nuttx-update
$ cd nuttx-update

## Repeat for all Conflicting Commits:
## Revert the Commit
$ git reset --hard HEAD~1
HEAD is now at 7d6b2e48044 gcov/script: gcov.sh is implemented using Python

## We have reverted One Conflicting Commit
$ git status
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.

## Push it to our Read-Write Mirror
## Our Read-Write Mirror is now back in sync with Read-Only Mirror!
$ git push -f
To nuttx-forge:nuttx/nuttx-update
e26e8bda0e9...7d6b2e48044 master -> master (forced update)

What if we really need to Accept Pull Requests in our Read-Write Mirror?

We’ll probably call GitHub API to ship the Pull Requests back to GitHub. (Pic below)

Right now we can’t allow Pull Requests to be locally merged at Forgejo. Instead, we’ll merge Pull Requests at GitHub. Then wait for Forgejo to auto-sync the updates back into our Git Forge.

Which explains why we need Two Mirror Repos: Read-Only Mirror and Read-Write Mirror.

§6 What’s Next

Next Article: Why Sync-Build-Ingest is super important for NuttX CI. And how we monitor it with our Magic Disco Light.

After That: Since we can Rewind NuttX Builds and automatically Git Bisect… Can we create a Bot that will fetch the Failed Builds from NuttX Dashboard, identify the Breaking PR, and escalate to the right folks?

Many Thanks to the awesome NuttX Admins and NuttX Devs! And My Sponsors, for sticking with me all these years.

• Sponsor me a coffee

• Discuss this article on Hacker News

• My Current Project: “Apache NuttX RTOS for Sophgo SG2000”

• My Other Project: “NuttX for Ox64 BL808”

• Older Project: “NuttX for Star64 JH7110”

• Olderer Project: “NuttX for PinePhone”

• Check out my articles

• RSS Feed

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.org/src/forgejo.md

§7 Appendix: Install our Forgejo Server

Here are the steps to install our own Forgejo Server (pic above) on Docker Engine…

(Derived from Official Docs)

(Tested on macOS Rancher Desktop)

## Download the Forgejo Docker Image
## And our Modified Docker Compose Config
docker pull codeberg.org/forgejo/forgejo:9
cd $HOME
git clone https://nuttx-forge.org/lupyuen/nuttx-forgejo
cd nuttx-forgejo

## docker-compose.yml: Points to `forgejo-data` as the Data Volume (instead of Local Filesystem)
## Because Rancher Desktop won't set permissions correctly for Local Filesystem (see secton below)
## services:
##   server:
##     volumes:
##       - forgejo-data:/data
## volumes:
##   forgejo-data:

## Up the Forgejo Server
## (Is `sudo` needed?)
sudo docker compose up

## If It Quits To Command-Line:
## Run a second time to get it up
sudo docker compose up

## Always restart Forgejo Server
## Whenever it crashes
sudo docker update --restart always forgejo

## If we need to down the Forgejo Server:
## sudo docker compose down

(See the Install Log)

(Fixing the Docker Compose Config)

• This will auto-create the forgejo folder for Forgejo Data

• We browse to http://localhost:3002

• Select SQLite as the database (we upgrade to PostgreSQL later)

• Set Domain to nuttx-forge.org (or your domain)

• Create an Admin User named nuttx (or your preference)

• Talk to our Web Hosting Provider (or Tunnel Provider).

  Channel all Incoming Requests for https://nuttx-forge.org

  To http://YOUR_DOCKER_MACHINE:3002

  (HTTPS Port 443 connects to HTTP Port 3002 via Reverse Proxy)

  (For CloudFlare Tunnel: Set Security > Settings > Low)

  (Change nuttx-forge.org to Your Domain Name)

• Remember to Backup Forgejo regularly!

  ## Inside Docker: Amalgamate the `/data` folder into `/tmp/data.tar`
  sudo docker exec \
    -it \
    forgejo \
    /bin/bash -c \
    "tar cvf /tmp/data.tar /data"
  
  ## Copy `/tmp/data.tar` out from Docker
  sudo docker cp \
    forgejo:/tmp/data.tar \
    .

  (See the Backup Log)

• Inside the Backup: SQLite Database is at data/gitea/gitea.db

  

Back to the Forgejo Configuration: This is how we specify the Forgejo Database…

And the Server Domain…

Finally our Admin User…

Forgejo’s Default Page: How to change it?

We copy out the Forgejo Configuration from Docker…

sudo docker cp \
  forgejo:/data/gitea/conf/app.ini \
  .

Edit app.init and set…

[server]
LANDING_PAGE = explore

Then copy it back to Docker…

sudo docker cp \
  app.ini \
  forgejo:/data/gitea/conf/app.ini
sudo docker compose down
sudo docker compose up

The Default Page for our Forgejo Server…

Now becomes a little more helpful…

(Popolon has plenty more tips for Forgejo Server)

§8 Appendix: Read-Only Mirror

Now that Forgejo is up: Let’s create a Read-Only Mirror of the NuttX Repo at GitHub.

Forgejo shall auto-sync our repo (every hour), but it won’t allow Pull Requests in our Read-Only Mirror…

1. At Top Right: Select + > New Migration

   

2. Select GitHub

   

3. Enter the GitHub URL of NuttX Repo

   Fill in the Access Token

   Check “This Repo Will Be A Mirror”, Migrate LFS Files and Wiki

   Set the Repo Name to nuttx-mirror

   

4. This will create a Read-Only Mirror: nuttx-mirror

   Forgejo won’t migrate the other items: Issues, Pull Requests, Labels, Milestones, Releases (pic above)

   (Read-Write Mirror will be more useful, see the next section)

5. And Forgejo dutifully creates our Read-Only Mirror!

   (Live Site)

   

6. By Default: Forgejo syncs every 8 hours. We change the Mirror Interval to 1 hour

   That’s under Settings > Repository > Mirror Settings

   

7. Forgejo has helpfully migrated our Template for NuttX Issues

   (Live Site)

   

8. Forgejo has ported over our GitHub Actions Workflows. But they won’t run because we don’t have a CI Server for Ubuntu x64.

   (Live Site)

   

9. NuttX Commits look very familiar in Forgejo

   Commit Hashes are identical to GitHub

   (Live Site)

   

10. So cool to watch Forgejo Auto-Sync our GitHub Repo

    (Live Site)

    

11. Auto-Sync may trigger CI Workflows. But we don’t have CI Servers to run them (yet).

    (Live Site)

    

12. That’s why the CI Jobs will wait forever

    (Live Site)

    

13. GitHub Reusable Workflows are Not Supported in Forgejo.

    This means the NuttX Build Rules (arch.yml) probably won’t run on Forgejo.

14. Git Command-Line Tools will work great with our Forgejo Server

    ## Download the NuttX Mirror Repo
    ## From our Forgejo Server
    git clone \
      https://nuttx-forge.org/nuttx/nuttx-mirror
    
    ## Also works for SSH (instead of HTTPS)
    ## But SSH isn't enabled on our server yet
    git clone \
      [email protected]:nuttx/nuttx-mirror

§9 Appendix: Read-Write Mirror

Earlier we created a Read-Only Mirror. But it doesn’t allow Pull Requests!

Now we create a Read-Write Mirror of the NuttX Repo at GitHub, which will allow Pull Requests. Forgejo won’t auto-sync our repo, instead we’ll run a script to sync the repo…

1. At Top Right: Select + > New Migration

   

2. Select GitHub

   

3. Enter the GitHub URL of NuttX Repo

   Fill in the Access Token

   Uncheck “This Repo Will Be A Mirror”

   Check the following: Migrate LFS Files, Wiki, Labels, Milestones, Releases

   Set the Repo Name to nuttx-update

   

4. This will create a Read-Write Mirror: nuttx-update

   Forgejo won’t auto-sync our repo. But it will migrate the other items: Labels, Milestones, Releases (pic above)

   Don’t select Issues and Pull Requests! Forgejo will hang forever, hitting errors. (Probably due to the sheer volume)

   

5. Assuming we didn’t select Issues and Pull Requests…

   Forgejo creates our Read-Write Mirror

   (Live Site)

   

6. How to sync the Read-Write Mirror? We run this script…

   sync-mirror-to-update.sh

§10 Appendix: Pull Request in Forgejo

How different are Forgejo Pull Requests from GitHub?

Let’s find out!

1. We create a Fork of our NuttX Read-Write Mirror

   (Live Site)

   

2. Create a new branch: test-branch. Edit a file in our new branch.

   (Live Site)

   

3. Save the file to our new branch

   

4. Click “New Pull Request”

   

5. Again click “New Pull Request”

   

6. Remember the NuttX Template for Pull Requests? It appears in Forgejo

   

7. Click “Create Pull Request”

   

8. And we’ll see our New Pull Request

   (Live Site)

   

9. Indeed, no surprises! Everything works the same.

   

10. Merging a Pull Request will trigger the exact same CI Workflow. Which won’t run because we haven’t configured the CI Servers.

    (Live Site)

    

11. Will Forgejo handle Large Pull Requests? Yep here’s a (potential) Pull Request with 217 NuttX Commits

    (Live Site)

    

12. Let’s try not to Merge any Pull Request into our Read-Write Mirror. We should keep it in sync with our Read-Only Mirror!

    “Sync our Read-Write Mirror”

§11 Appendix: SSH Access in Forgejo

This section explains how we tested SSH Access in our Forgejo Server.

Note: SSH Port for our Forgejo Server is not exposed to the internet (for security reasons).

We generate the SSH Key…

$ ssh-keygen -t ed25519 -a 100
## Save it to ~/.ssh/nuttx-forge

Edit ~/.ssh/config and add…

Host nuttx-forge
  HostName localhost
  Port 222
  IdentityFile ~/.ssh/nuttx-forge

(localhost will change to the External IP of our Forgejo Server)

In Forgejo Web:

• Click User > Settings > SSH Keys
• Paste the contents of ~/.ssh/nuttx-forge.pub
• Click Add Key

Finally we test the SSH Access…

$ ssh -T git@nuttx-forge  

Hi there, nuttx! You've successfully authenticated with the key named nuttx-forge (luppy@localhost), but Forgejo does not provide shell access.
If this is unexpected, please log in with password and setup Forgejo under another user.

We create a Test Repo in our Forgejo Server…

And we Commit over SSH to the Test Repo…

git clone git@nuttx-forge:nuttx/test.git
cd test
echo Testing >test.txt
git add .
git commit --all --message="Test Commit"
git push -u origin main

We should see the Test Commit. Yay!

§12 Appendix: SSH vs Docker Filesystem

Why did we change the Docker Filesystem for Forgejo?

Based on the Official Docs: Forgejo should be configured to use a Local Docker Filesystem, ./forgejo

## Officially: Docker mounts Local Filesystem
## `./forgejo` as `/data`
services:
  server:
    volumes:
      - ./forgejo:/data

Let’s try it on macOS Rancher Desktop and watch what happens…

## Connect to Forgejo Server over SSH
ssh -T -p 222 git@localhost

Forgejo Server Log says…

Authentication refused: 
  bad ownership or modes for 
  file /data/git/.ssh/authorized_keys

Connection closed by authenticating
  user git 172.22.0.1 port 47768 [preauth]

We check the SSH Filesystem in macOS…

$ ls -ld $HOME/nuttx-forgejo/forgejo/git/.ssh                
drwx------@ 4 luppy  staff  128 Dec 20 13:45 /Users/luppy/nuttx-forgejo/forgejo/git/.ssh

$ ls -l $HOME/nuttx-forgejo/forgejo/git/.ssh/authorized_keys 
-rw-------@ 1 luppy  staff  279 Dec 21 11:13 /Users/luppy/nuttx-forgejo/forgejo/git/.ssh/authorized_keys

Do the same Inside Docker…

## Connect to the Docker Console
$ sudo docker exec \
  -it \
  forgejo \
  /bin/bash

## Check the SSH Filesystem inside Docker
$ ls -ld /data/git/.ssh
drwx------    1 501  dialout  128 Dec 20 13:45 /data/git/.ssh

$ ls -l /data/git/.ssh/authorized_keys
-rw-------    1 501  dialout  279 Dec 21 11:13 /data/git/.ssh/authorized_keys

Aha! User ID should be git, not 501! (Some kinda jeans?)

Too bad chown won’t work…

## Nope! Won't work in Rancher Desktop
exec su-exec root chown -R git /data/git/.ssh

Sadly, macOS Rancher Desktop won’t set permissions correctly for the Local Filesystem.

And that’s why our docker-compose.yml points to Data Volume forgejo-data (instead of Local Filesystem)

## Our Tweak: Docker mounts Data Volume
## `forgejo-data` as `/data`
## (Instead of Local Filesystem `./forgejo`)
services:
  server:
    volumes:
      - forgejo-data:/data

## Declare the Data Volume `forgejo-data`
volumes:
  forgejo-data: