Building an x86-64 Operating System from Scratch in C: The Exploidus Story

Leader ●1 ●2 ●8
calendar_today ago • schedule6 min read
— Originally published at dev.to

Most software engineers spend their careers working high up the abstraction ladder: web frameworks, containers, virtual machines, and cloud APIs. We treat the operating system as an invisible, infallible foundation. We assume malloc() will magically allocate RAM, write() will reliably flush bytes to disk, and the CPU will seamlessly multiplex threads.

A few years ago, I decided to pull back the curtain and ask a fundamental question:

What actually happens if you strip away Linux, Windows, libc, and every single runtime library, and sit directly on the bare metal of an x86-64 processor with nothing but C and assembly?

That question led to Exploidus—a custom x86-64 reactive capability operating system kernel built completely from scratch.

Today, Exploidus boots via Multiboot2/GRUB, runs in full 64-bit long mode, implements 4-level paging with NX enforcement, features a custom journaling filesystem (ExFS), runs an in-kernel TCP/IP stack with real Gigabit Ethernet drivers, enforces BLAKE3 cryptographic capability tokens, supports 82 POSIX-style syscalls, and even powers a double-buffered userspace window compositor.

Here is the story of how I designed and built it, the architectural hurdles I faced, and what bare-metal systems programming teaches you about software engineering.

  1. The Starting Line: From Real Mode to 64-Bit Long Mode
    When an x86-64 processor powers on, it does not wake up as a modern 64-bit machine. It awakens in 16-bit Real Mode, essentially pretending to be an Intel 8086 from 1978 with 1 MB of addressable memory.

To get Exploidus running, the bootloader transition had to be orchestrated meticulously:

Multiboot2 Compliance: The kernel binary begins with a Multiboot2 header so GRUB2 can discover and load our ELF64 image into memory.
Transitioning Modes: Moving through 32-bit Protected Mode, setting up the Global Descriptor Table (GDT), enabling Physical Address Extension (PAE), and finally setting the Long Mode Enable (LME) bit in the EFER model-specific register.
Entering Long Mode: Once paging is turned on and a 64-bit GDT code descriptor is loaded, a far jump lands execution into pure 64-bit Long Mode.
At this stage, you have no printf, no standard library, and no memory allocator. If a pointer dereferences wrong, the CPU triple-faults and the machine instantly reboots in a split second.

The first milestone of any OS developer is the VGA text-mode driver at physical address 0xB8000. Writing characters directly into video memory bytes and watching the word Exploidus Kernel Initializing... render on a blank screen is a sensation every programmer should experience at least once.

  1. Memory Architecture: 4-Level Paging and Colored Zones
    Virtual memory is the bedrock of process isolation. Without it, any runaway pointer can overwrite the kernel or another process's state.

In Exploidus, memory management is divided into two core layers:

4-Level x86-64 Paging
Exploidus uses the standard x86-64 page-table hierarchy:

PML4 (Page Map Level 4)
PDPT (Page Directory Pointer Table)
PD (Page Directory)
PT (Page Table)
Every page entry manages a 4 KB physical frame. To enforce strict security:

NX (No-Execute) Bit Enforcement: All data, heap, and stack pages are flagged with the XD/NX bit (bit 63). Code cannot execute from the stack or heap, neutralizing classic buffer overflow shellcode attacks at the hardware level.
User/Supervisor Protection: Kernel space is mapped into the higher half and strictly protected by disabling the User bit (U/S = 0), preventing Ring 3 userspace applications from reading or writing kernel data structures.
Colored Physical Frame Allocation
Instead of a naive linear allocator or a slow buddy system, Exploidus introduces a Colored Zone Physical Memory Manager:

GREEN ZONE: Unrestricted, verified general-purpose memory for userspace processes.
YELLOW ZONE: DMA-capable and peripheral-bounded memory mapped for hardware device drivers (e.g. NIC ring buffers, USB host controllers).
RED ZONE: Cryptographically isolated, non-swappable physical frames reserved strictly for kernel capability tokens, master page tables, and audit logs.

  1. Capability Security & Crash Isolation
    Most commodity operating systems rely on ambient authority (e.g. standard UID/GID or Unix permissions). If a process runs as root, it can execute anything.

Exploidus explores a Reactive Capability-Based Security Model:

Resources (file descriptors, sockets, physical devices, IPC channels) are gated behind unforgeable capability tokens.
Hardware-Seeded BLAKE3 Tokens: Tokens are generated by hashing resource pointers and generation counters using the BLAKE3 cryptographic algorithm, seeded with random entropy directly from the CPU's RDRAND instruction.
Crash Isolation: If a userspace process suffers a segmentation fault, divides by zero, or triggers an invalid opcode, the kernel does not panic. The CPU interrupt handler traps the exception, destroys the faulting process's capability table, reclaims its address space, and terminates only that process. The shell and the rest of the OS continue running uninterrupted.

  1. Designing a Custom Filesystem: ExFS
    You cannot have a functional operating system without persistent storage. While many hobby kernels stop at read-only tar ramdisks or FAT12, Exploidus features its own custom filesystem: ExFS.

Key Architectural Elements of ExFS:
Generic Block Device Layer: An abstraction layer (blockdev.c) decouples storage hardware from the filesystem. Whether a volume is an IDE/ATA hard drive (ata0) or a USB thumb drive (usb0), ExFS interacts through unified sector read/write interfaces.
Metadata Journaling: ExFS uses a write-ahead metadata journal to ensure crash consistency. Directory modifications, inode allocations, and block allocations are recorded in a circular journal before being committed to disk structures. If a sudden power loss occurs, replay recovery restores volume consistency upon reboot.
Data Provenance: Inodes in ExFS carry immutable provenance records, tracking the creator capability and timestamp of creation to provide built-in forensic audit trails.

  1. Drivers from the Metal: USB and Gigabit Ethernet
    Writing device drivers without third-party libraries or Linux helper APIs is where hardware reality hits you hardest. Data sheets for real silicon can be hundreds of pages long and filled with subtle timing quirks.

UHCI USB Subsystem
Exploidus includes a custom UHCI (Universal Host Controller Interface) USB driver:

Probes PCI configuration space to discover USB host controllers.
Configures physical memory transfer descriptors (TDs) and queue heads (QHs) in DMA-accessible memory.
Implements USB device enumeration, Control, Interrupt, and Bulk endpoints.
Supports USB Mass Storage (Bulk-Only Transport + SCSI command wrappers), allowing developers to plug in a USB flash drive, mount it into the VFS, and read files via the shell.
Gigabit Networking & In-Kernel TCP/IP Stack
Exploidus features an in-kernel network stack communicating through an Intel 82540EM (e1000) Gigabit Ethernet controller:

Layer 2: Ethernet frame parsing and ARP table resolution with cache expiration.
Layer 3: IPv4 parsing, checksum calculation, and IP fragment reassembly.
Layer 4: ICMP (ping echo reply/request), UDP, and full TCP state machine (three-way handshake SYN -> SYN-ACK -> ACK, sequence/acknowledgment tracking, window scaling, and packet retransmission).
Because the TCP/IP stack runs natively inside the kernel, userspace processes can issue POSIX-like socket(), connect(), send(), and recv() syscalls to communicate over the real local network.

  1. From the exploish Shell to a GUI Compositor
    An OS needs an interface to be useful. Exploidus provides two layers:

The exploish Interactive Shell
exploish runs as a Ring 3 userspace process, executing commands via system calls:

ps: Inspects real kernel process tables, priorities, and execution states.
audit: Reads cryptographic audit log entries directly from the kernel.
mount / ls / cat: Traverses and manipulates the ExFS filesystem.
The alien Window Compositor
Moving beyond text mode, Exploidus includes a userspace graphical compositor called alien:

Utilizes VESA / Linear Framebuffer modes discovered via Multiboot.
Single-Syscall Window Blitting (SYS_FB_BLIT): Minimizes user-to-kernel context switching during rendering.
Double-Buffering & Dirty-Region Redraws: Windows maintain their own back-buffers in userspace; the compositor only copies modified bounding boxes to the screen buffer, providing flicker-free rendering with real-time frame pacing.

  1. What Building an OS Taught Me
    Writing an operating system from scratch over months and years completely reshapes how you look at software engineering:

Hardware is not clean; it is quirky: Silicon behaves in strange ways. Concurrency is not just threads racing; it is hardware interrupts firing in the middle of a critical pointer assignment. You develop an immense respect for atomic operations, IRQ-safe spinlocks, and memory barriers.
Abstractions have a real cost: When you have to write your own kmalloc(), page fault handler, and scheduling timer, you realize that memory allocations and context switches are not free. High-level frameworks often hide enormous amounts of churn.
Debugging without tools builds discipline: When you don't have GDB or stack traces, and an invalid memory write freezes the CPU instantly, you learn to reason through your code with mathematical precision. You read every line, analyze register states, and build defensive invariants.
Conclusion & Open Source
Exploidus started as an ambitious experiment in Bangladesh to explore what it takes to build an operating system from zero without relying on existing Linux codebases. Today, it stands as a fully documented, open-source x86-64 capability kernel with a rich ecosystem of drivers, filesystems, and networking.

If you are interested in OS development, systems programming, or seeing how an x86-64 kernel works under the hood, the entire codebase is open-source:

GitHub Repository: https://github.com/rahadbhuiya/Exploidus
Feel free to star the repo, clone it, run it in QEMU, or read through the kernel source!

Have you ever experimented with operating system development or bare-metal programming? What was the hardest low-level bug you ever encountered? Let's discuss in the comments below!

🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

Your Tech Stack Isn’t Your Ceiling. Your Story Is

Karol Modelski - Apr 9

TypeScript Complexity Has Finally Reached the Point of Total Absurdity

Karol Modelski - Apr 23

Frameworks Are Institutional Memory

Ken W. Algerverified - Sep 17

I’m a Senior Dev and I’ve Forgotten How to Think Without a Prompt

Karol Modelski - Mar 19

MCP Is the USB-C of AI. So Why Are You Plugging Everything In?

Ken W. Algerverified - Jun 10
chevron_left
798 Points • 11 Badges
3Posts
2Comments
3Connections
Systems programmer & open-source builder.

Related Jobs

View all jobs →

Commenters (This Week)

3 comments
1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!