Like many dynamic languages out there, lone started out simple. It was essentially a collection of data structures written in C, an union comprising all of those types, a typed value structure containing the union plus metadata, and a custom language whose entire purpose is to bring all these values together into patterns that resemble working programs.

struct lone_lisp_list   { /* ... */ };
struct lone_lisp_vector { /* ... */ };
struct lone_lisp_table  { /* ... */ };
/* ... */

enum lone_lisp_heap_value_type {
    LONE_LISP_TYPE_LIST,
    LONE_LISP_TYPE_VECTOR,
    LONE_LISP_TYPE_TABLE,
    /* ... */
};

struct lone_lisp_heap_value {
    bool live: 1;
    bool marked: 1;
    /* ... */

    enum lone_lisp_heap_value_type type;

    union {
        struct lone_lisp_list list;
        struct lone_lisp_vector vector;
        struct lone_lisp_table table;
        /* ... */
    } as;
};

When I put it this way, the language itself seems almost incidental to the virtual machine work. That's because it is. Lone was in fact not designed ahead of time. It was and still is being constructed in real time. It's almost as if the language itself is arising as a consequence of its own implementation. It grows in complexity alongside the knowledge and understanding of its creator.

At first I was a neophyte. I was attracted to ideas that were almost painfully simple, almost too naive to cope with the harsh reality of the world. The code reflected that.

Take the above structures, for example. How are such values created? My first instinct is to simply allocate some memory for each object. Call malloc with the size of the structure, then initialize the memory it returns. It's that easy.

Right?

Memory allocation

Lone is a lisp interpreter written in freestanding C. There is no dynamic memory allocation in freestanding C. There is no such thing as malloc. There is no libc. There is only me and the code. If I wanted malloc, I would have to write it myself.

So I wrote it. I read up on everything I could find online about memory and its allocation. Then I made my own memory allocator.

To manage a memory block, the allocator needs to describe it. Let's start with the basics. The most basic information about a piece of data is its location and its size.

struct lone_memory {
  size_t size;
  unsigned char pointer[];
};

The allocator must also keep track of whether the block is free or currently in use. Otherwise, there would be chaos.

struct lone_memory {
  bool free;
  size_t size;
  unsigned char pointer[];
};

This is enough to manage any one block of memory. If it's free, then the allocator can give the block to whoever asks. If it's not free, then it can't. If they ask for less or exactly as much memory as this block contains, then it can be given out. If they ask for more, then it can't.

I started from literally nothing. Now I've got the one. Time to handle infinity.

struct lone_memory {
  struct lone_memory *prev, *next;
  bool free;
  size_t size;
  unsigned char pointer[];
};

Simply link the blocks to each other. Now if some code requests a block, the allocator can walk through the list of all blocks and search for any block that fits.

And that's exactly what the allocator does.

for (block = system->memory; block; block = block->next)
    if (block->free && block->size >= size)
        break;

if (!block) { return 0; }

block->free = false;
return block->pointer;

Searches the list of blocks and literally returns the first block that fits. This thing could end up returning 16 KiB blocks for 64 byte requests. Crude, but effective. Incredibly wasteful, of course. Potentially more wasteful than mmaping pages for every single allocation.

The memory allocator can do better than this. Why not cut the block up into smaller chunks?

block->free = false;
lone_memory_split(block, size);
return block->pointer;

Split the block into two blocks: an allocated block of exactly the requested size, and a new free block for any excess memory that might remain.

size_t excess = block->size - size;

/* create a new block only if there's enough
   space for memory block descriptor + 1 byte */

if (excess >= sizeof(struct lone_memory) + 1) {
    new = (struct lone_memory *) (block->pointer + size);

    /* weave the new block into the linked lists */

    new->free = true;
    new->size = excess - sizeof(struct lone_memory);
    block->size = size;
}

The excess memory block gets conjured up out of thin air: the allocator simply drops a new memory block descriptor right after the end of the previous memory block. When the links are established, the excess memory can be allocated like any other memory block.

Memory deallocation is even simpler: just mark the block as free. The block descriptor is just behind the pointer, trivially reachable.

struct lone_memory *block = ((struct lone_memory *) pointer) - 1;
block->free = true;

This is enough, but the allocator can do better. It can check if surrounding blocks are also free, and undo the split if that is the case.

lone_memory_coalesce(block);
lone_memory_coalesce(block->prev);

Coalescing a memory block is simple. When two adjacent blocks are free, one block literally absorbs the other, block descriptor and all.

if (block && block->free) {
    next = block->next;
    if (next && next->free) {
        block->size += next->size + sizeof(struct lone_memory);
        /* unlink the blocks */
    }
}

That's it. A simple yet complete first fit, split/coalesce memory allocator. Give it a large block of memory to manage and watch as it cuts the block up into small chunks and hands them all out to anyone who asks.

#define LONE_MEMORY_SIZE (64 * 1024)
static unsigned char memory[LONE_MEMORY_SIZE];

lone_lisp_initialize(&lone, memory, sizeof(memory));

How good is this memory allocator? Let's just say that terrible doesn't quite do it justice. This thing will linearly scan the list of blocks for every single allocation. It will fail pretty hard at keeping memory fragmentation under control, which not only wastes memory but could result in entirely avoidable out-of-memory situations. Small leftover blocks tend to accumulate near the start of the list, further compounding the problems as it runs. Prefixing block descriptors to every block means metadata overhead of around 30% to 50% for typical allocations, not to mention the fact those headers are classic targets for exploitation. The shortcomings of this allocator could fill up an entire article all by themselves.

Nevertheless, lone made do with this thing for around three years. For all of its flaws, it absolutely does allocate memory, and memory was all that I needed to make some objects. I didn't really care about the allocator, I just needed some values I could play with as I developed the language.

Yeah, I'll totally go ahead and do that now. Allocate some values, I mean. Everything's gonna be simple now.

Right?

Scanning for pointers

It's not that simple. Other parts of the code have requirements that must be fulfilled.

The lone lisp garbage collector is conservative. It walks the stack and scrutinizes everything it finds. It wants to know whether they are pointers to lisp objects. In order to do that, it must maintain a list of every single lisp value pointer.

Sounds simple when I put it that way. Why not just do it? Just scan the stack and compare every single word against every single lisp pointer, right? Right?!

Yeah, no. That's not going to happen. Well, not unless you want to get into the quadratic hall of shame. There's got to be a way, some clever data structure to make the problem go away...

The first heap

Wait, I think I know what to do. During my tenure in the Ruby mines many eons ago, I just so happened to pick up some tricks. That includes this one:

struct lone_lisp_heap {
    struct lone_lisp_heap *next;
    struct lone_lisp_heap_value values[LONE_LISP_HEAP_VALUE_COUNT];
};

A linked list of arrays of values!

General purpose memory allocators provide no guarantees; those must be forged by hand by way of data structures. Now objects aren't individually allocated any more, the language buys them in bulk. The lone heap is essentially a list of nice little chunks of lisp objects, all lying right next to each other with zero gaps in between them. Now the garbage collector can simply check if pointers fall within range and call it a day. Alright!

Mechanically, it's almost identical to the general purpose memory allocator. It just works at the lisp value level instead of working at the byte level. There's a linked list of chunks, it walks through them all. For each chunk, it loops over the values. If it finds a dead value, it just returns it. Chunks are allocated with all values dead. The garbage collector just kills the values in those slots instead of deallocating them.

The allocator isn't really allocating values, it's resurrecting them.

The power of arrays never fails to impress. It's literally just juxtaposing things together. Sounds like something a stupid caveman would do. Grug programmer slaps things together into arrays while the refined gentleman weaves linked lists with grace and finesse. Yet it's arrays which the processor likes, and it's arrays that the computer science validates as the good solution to this problem. Go figure. It almost makes me want to use arrays for literally everything.

Can I do that? Use one big stupid flat array for literally everything? Is it even possible?

The tyranny of the pointers

The trouble with these arrays is precisely the fact they're big stupid monolithic chunks of objects. Can't easily insert new values in the middle of chunks, that would require resizing the array. Easier to just put them into a linked list and call it a day. That way if more storage is needed one can just allocate new chunks and link them in, and if the entire chunk becomes dead one can just link them out and deallocate the chunks. The insertion performance of linked lists combined with the awesomeness of arrays. This must be what peak performance looks like.

But why is that, anyway? Why is it impossible to easily resize these arrays? Things would be so much easier if we could do that...

I kept thinking about it, then I realized there was no such thing as "resizing" an array to begin with. That was an abstraction, a fantasy. In reality, a completely new array gets allocated, the old data gets copied over, and then the old array gets destroyed. That's what array "resizing" actually is. It's destructive reconstruction. Like wars and the economy.

A completely new array is allocated every single time. This array could be in a totally different location. This invalidates all pointers into the array. All lone lisp objects are pointers to heap values inside these arrays. Resizing them would dangle every pointer to the values inside them. It would be pure chaos.

Such is the tyranny of the pointers. They are lazy and unmoving, eternally fixed in place, utterly unconcerned about the evolution of the world around them. They are the NIMBYs of the programming world. Entire new edifices of memory are getting built all around them, yet they do not react. They do nothing but drag their feet and remain where they are until the world ends, forcing the rest of the world to adapt to them instead. Yes, it is clear to me now. Someone would have to do something about all of these pointers.

But what exactly should be done about them? Pointers aren't gonna change. Not for me, not for you, not for anyone tonight. If anything was going to change, it was going to have to be the world. Could I change the world?

What if the values... Weren't pointers to begin with? What if they were just... Numbers? Indexes into the array? Easier said than done. Like many lisps before it, lone used a tagged pointer representation. In lisp, everything is kung fu pointers. Such a fundamental change would touch everything in the interpreter.

Did it anyway. Rewrote the entire value representation. Mercifully, it had already been abstracted away into nice functions and structures by this point. This was not the first time it was rewritten, or even the second. Won't be the last time either. Mark my words.

So now lone values are no longer pointers, they are indexes into the lone value heap. Yes, the lone heap. Singular. The linked lists are gone now.

struct lone_lisp_heap_value *
lone_lisp_heap_value_of(struct lone_lisp *lone, struct lone_lisp_value value)
{
    size_t index = /* get the index out of the value */;

    return &lone->heap.values[index];
}

The heap is literally just a big dumb flat array of values now. A black monolith. An absolute unit. Lone values simply index into this array. The array could be anywhere in memory, it doesn't matter where it is. The real pointers are simply calculated on the fly.

Values are now independent of their position in memory. Lone can now fearlessly reallocate, move and resize the heap.

mremap

A lot of things are falling into place here all at once.

Position independent values. One big dumb array of values. Why not just... Memory map this array as a huge page? Lone's heap values are currently 56 bytes. Let's bump that up a bit by aligning it to 64 bytes. This means they will never ever overlap page boundaries, no matter what page size Linux uses. And would you look at that... 64 bytes just happen to be exactly one cache line.

Lone's heap is made out of pages now. It is literally just pages and pages and more pages, all filled to the brim with lisp values. From byte allocation to value allocation to page allocation. Not bad...

void lone_lisp_heap_initialize(struct lone_lisp *lone)
{
    size_t size = LONE_LISP_HEAP_CAPACITY
                * sizeof(struct lone_lisp_heap_value);

    lone->heap.values = linux_mmap(0, size,
                                   PROT_READ   | PROT_WRITE,
                                   MAP_PRIVATE | MAP_ANONYMOUS,
                                   -1, 0);

    lone->heap.count = 0;
    lone->heap.capacity = LONE_LISP_HEAP_CAPACITY;
}

mmap is awesome but Linux's got something even more awesome: mremap, which makes it almost trivial to manage the page-based heap. The kernel can restructure page tables at will, which means it can move the entire lone heap without copying anything! The man page wasn't kidding about the fact that it could be used to implement a very efficient realloc.

static void lone_lisp_heap_grow(struct lone_lisp *lone)
{
    size_t old_capacity, new_capacity,
           old_size,     new_size;

    old_capacity = lone->heap.capacity;
    old_size = old_capacity * sizeof(struct lone_lisp_heap_value);

    new_capacity = old_capacity * 2;
    new_size = new_capacity * sizeof(struct lone_lisp_heap_value);

    lone->heap.values = linux_mremap(lone->heap.values,
                                     old_size, new_size,
                                     MREMAP_MAYMOVE);

    lone->heap.capacity = new_capacity;
}

The lone heap

From nothing to a specialized mremappable cache friendly heap of lone lisp values. Not bad for a homegrown programming language.

It's still linearly scanning for dead values to resurrect though. Starts from the beginning every single time. It has to. The garbage collector could theoretically choose to assassinate any object in the program, including the one at index zero. It can't assume that the zeroth object will always be alive.

Right?

Thirteen years ago, in two thousand thirteen, Baby’s First Garbage Collector was born. Now he's grown up. He's outgrown his shallow kiddie pool and his paddles. He's entering puberty. Soon he'll be in college. They grow up so fast.

Bob Nystrom wasn't kidding about Baby's First Garbage Collector. It absolutely does collect garbage, and a version of it absolutely did ship with my dynamic language, lone lisp. In fact, if I remember correctly, lone actually still ships with it, believe it or not. He wasn't kidding about communion with the Elder Gods either. I can feel my magic level increasing. Wizardry is within reach.

Baby's First Garbage Collector is a starting point though. It's not meant to stop there. It's meant to grow into something better and more complex as it morphs around the actual language it supports. It's meant to reach its final form. Perfectly Ultimate Great Garbage Collector.

It'll be quite a few turns before that happens but chronicling its evolution is still going to be awesome. It's coming along nicely.

Trouble in the primitive lands

Baby's First Garbage Collector is what is called a precise garbage collector. It knows where all the objects are at all times. It knows where they live. It engages in Orwellian surveillance of those objects. It is always ready to stop the world and reap those objects the second they step out of the stack. It oppresses the objects with such precision and automation, it is impossible for even a single one of those objects to escape this fate. Thus it rules the objects with a silicon fist, quite literally deciding which objects live or die, and even when they die. Such is the life of an object in the dynamic lands...

It was only a matter of time before a resistance formed. The poor objects wanted freedom from the tyranny of the garbage collector. They needed to find a way to escape. But how? Where could they go? They didn't know. They just knew they had to escape the stack.

LONE_LISP_PRIMITIVE(run_objects_run)
{
    struct lone_lisp_value object;

    object = lone_lisp_machine_pop_value(lone, machine);

    /* freedom */
}

There they go. They are free now. Free of the reaping machine's clutches. Free to enjoy their lives as they see fit, until the very end of that function. For those ten microseconds or less, they're free.

Their happiness is short-lived. You see, the garbage collector has foreseen this and is well prepared: it has implanted every single object with a tracker since the day they were born, the easier to hunt them down with.

The garbage collector maintains a census: a list of every single object who ever lived. It can reach them through this artifice. So they might just discover that the reaper will morph into existence right in front of them out of nowhere like Agent Smith and reap their souls when they least expect it.

The garbage collector doesn't do this to just anybody. It only does this to garbage. It only does this to the undesirables. The problem is when those objects escaped and went on to enjoy their well-earned freedom, they became garbage in the garbage collector's eyes. The reaper tried calling them at work and they didn't answer. So it set about hunting them down. Some of these objects were perfectly good members of society. They would have gone back to the program eventually. They got reaped all the same...

The problem is the garbage collector is not as omnipresent as was once believed. There are nooks and crannies it won't check. There are places it can't reach. Primitive places, magical fissures where the world intertwines with a hellish and unexplored dark realm only wizards dare venture into. Objects can hide in there, beneath the garbage collector's notice, a mistake that proves fatal for them in the long run, a mistake that leads to the utter destruction of dynamic society. The machine cares not where they hide, it will find them and it will reap them even if it must go to the depths of hell itself to do it. However, the civilian casualties of this war on garbage are mounting. The collector doth collect too much, and without those objects, the very fabric of the program will unravel in short order.

Into the nether realms

The elder gods who created the garbage collector would not stand for this. It was forced to grow. It was forced to evolve. It was forced to search the underworld for the lost children that escaped it and bring them back safely, that the program may continue uninterrupted by nonsensical crashes and unwindings.

The dark mages often braved the underworld themselves and were therefore undaunted by the task. It should not be difficult, they thought, to adapt the machine to do it. Why couldn't it travel the foreign lands? There was no reason. And so it was decided. The machine would be taught how to do it.

The resources available at the garbage collector's disposal were substantial. It had the object census. It had a list of roots which it would search for objects. It would reap all objects it didn't find in those roots.

One of those roots is the lisp stack. As the program churns, values are placed in stasis and stored there so that they may be recovered later when needed. It is when they escape from this stack that they create havoc in dynamic society. But where are they escaping to?

The native stack

It turns out the underworld has a stack of its own. This fact was well known to the dark mages but was once thought inconsequential to the workings of the garbage collector. How they were proven wrong!

The native stack is just like the lisp stack, yet unfathomably different. The things that lurk there are known only to the engineers of the great compilers, and to those whose mental fortitude allows them to parse and understand the sacred tomes of platform ABIs. Even the elder gods knew better than to disrespect such texts, for the consequences are undefined.

It turned out that the machine did not actually need to understand the internals of the native stack. It needed only to understand its own objects. It was the finder of lost children. Its job was to look for them. Nothing else mattered. However, the shadow of the 64-bit address space is vast. It couldn't just start from anywhere. It had to be told where to start.

Way back when the universe was formed, the first stack frames came into existence. Nobody truly knows who mapped them there, though among the well-learned, whispers of a great kernel are heard, a certain "Linux".

It is known, however, that the program that executes beneath the dynamic reality eventually reaches lisp land. Beyond this point, no lisp object could ever travel, for the environment was far too hostile for them. Many debugging expeditions were mounted to determine the exact point where this occurred. Many objects gave their lives for this knowledge. To waste this opportunity would be to let it all have been for nothing, a truly unforgivable crime.

The exact value of the stack pointer could be divined. Certain secret incantations allowed the mysterious compilers to imbue a value with the frame address at the time of the ritual. One had to simply be at the right place at the right time.

long lone(int argc, char **argv, char **envp, struct lone_auxiliary_vector *auxv)
{
    void *stack = __builtin_frame_address(0);

    /* interpreter runs... */

    return 0;
}

That forbidden spell tipped the balance. The limits were set. Now all that remained for the garbage collector to do was search. It would start from wherever in the program it was and go all the way up to the frontiers of the lisp lands, never once faltering.

static void lone_lisp_mark_native_stack_roots(struct lone_lisp *lone)
{
    lone_lisp_mark_native_stack_roots_in_range(
        lone,
        lone->native_stack,
        __builtin_frame_address(0)
    );
}

Close examination of the stack pointer revealed that it was aligned to some power of two, as many such things often are. They must be, lest they be cursed with slowdown and exceptions. This was useful though, since it allowed the transparent reinterpretation of the native stack contents. The garbage collector needed not care what was actually there, it needed only to determine if it was one of its objects.

static void lone_lisp_mark_native_stack_roots_in_range(struct lone_lisp *lone, void *bottom, void *top)
{
    void *tmp, **pointer;

    /* an old trick from the Ruby mines */
    if (top < bottom) {
        tmp = bottom;
        bottom = top;
        top = tmp;
    }

    for (pointer = bottom; pointer < top; ++pointer) {
        if (lone_lisp_points_to_heap(lone, *pointer)) {
            lone_lisp_mark_heap_value(lone, *pointer);
        }
    }
}

And that, as they say, was it. It wasn't perfect, few things in life ever are, but it was enough. The garbage collector would loop through all the words on the stack, looking for pointers inside its own heap, pointers to the objects that ran away. And it would find them. And then it would leave them alone, knowing they were safe out there in the outskirts of the lisp plains.

The native stack was treacherous. It would often fool the garbage collector with mirages of objects long gone. It had no choice but to believe the lies, for only those enlightened by the long-lost knowledge of the type system of the native program, software whose compilation took place eons ago, would have a discerning enough eye to tell the difference between phantoms and real objects. This was an acceptable inefficiency. It kept dead objects alive... But not once did it lead to the death of living objects. Yes. It was enough.

The heap pointers

The garbage collector handled the values it found in the most curious of ways. The reaping ritual requires that every lisp value pointer be checked for reachability. Ill-advised programmers would naively attempt to check every lisp pointer against every other pointer in the native stack, leading to quadratic behavior which in turn yields nothing but disgrace and humiliation.

Not the garbage collector. It was better than that. It was prepared. It had carefully scouted the memory plains until it had compiled the addresses of all values. Careful study of their living arrangements revealed a very useful fact that would no doubt be mercilessly weaponized against them: they lived next to each other in a structure they often referred to as the heap.

struct lone_lisp_heap {
  struct lone_lisp_heap *next;
  struct lone_lisp_heap_value values[LONE_LISP_HEAP_VALUE_COUNT];
};

The garbage collector didn't need to check each pointer individually, it could simply check whether it pointed into each heap. Their numbers were manageable.

static bool lone_points_within_range(void *pointer, void *start, void *end)
{
    return pointer >= start && pointer < end;
}

static bool lone_lisp_points_to_heap(struct lone_lisp *lone, void *pointer)
{
    struct lone_lisp_heap *heap;

    for (heap = lone->heaps; heap; heap = heap->next) {
        if (lone_points_within_range(pointer, heap->values, heap->values + LONE_LISP_HEAP_VALUE_COUNT)) {
            return true;
        }
    }

    return false;
}

And just like that, lone has a conservative garbage collector. Until next time. Soon I'll write about—

The shark attacks

Running the test suite unleashed total pandemonium upon the lisp city. Objects were swept off their usual functions and replaced with totally different objects seemingly at random. The function that the interpreter's evaluator type checked suddenly turned into a hash table before application of arguments, a table that was supposed to have been allocated somewhere else. Tracing the interpreter through the debugger didn't make sense because the values that were just printed were suddenly becoming something else due to action at a distance. This can only mean one thing.

The task was not done yet. Some objects were still hiding.

The garbage collector had found most of them but about one or two dozen were still missing. Where could they be? There was only one place they could possibly be hiding...

The registers

When in doubt, one must read the ancient source tomes written by those who came before us. People like Hans Boehm, Alan Demers and Mark Weiser, who once upon a time also sat down and decided to write a garbage collector. For C and C++. Yes, you read that right.

I had to know. I had to know how they did it. The source code proved somewhat impenetrable, too dense to just dive in... However, after some exploration, I struck gold.

The porting guide mentioned a function named GC_with_callee_saves_pushed whose purpose is to spill the registers on the stack so that the garbage collector may trace it. And how was it implemented?

/* Generic code. */

jmp_buf regs;

/*
 * `setjmp()` does not always clear all of the buffer.
 * That tends to preserve garbage.  Clear it.
 */
BZERO(regs, sizeof(regs));

(void) setjmp(regs);

setjmp, of all things.

The porting guide frowns upon the assembly of one's own instructions. The architectural code is a form of black speech. Its words cannot be pronounced. Its sentences cannot be uttered. Warlocks have gone mad trying to decipher the mnemonics. To practice it is to partake in forbidden techniques abstracted away a long time ago, relevant only to the optimizers that were banished to the lower levels, where they lurk to this very day.

So it was not strange when the code dutifully reached into the standard library of scrolls for a spell which could produce the desired side effect. It did not matter what the spell had been created for. The elders had observed its workings and determined that it was all but adequate for the purpose at hand. They warned of preconditions for the success of the artifice, but admitted that even they were unsure of the steps necessary to complete the ritual.

I had to know. The answer had to be in setjmp.

Spilling the registers

The function came from the standard library, a dusty collection of spells developed by ancestors long gone, their magic often lost to the newer generations untainted by curiosity.

After exploring some more source code, I received enlightenment. I understood. I knew exactly what it was that I had to do.

I prepared myself with study, then entered the x86_64 realm. With my editor's cursor, I conjured into existence an avalanche of mov instructions.

typedef long lone_registers[16];
extern void lone_save_registers(lone_registers);

__asm__
(
    ".global lone_save_registers"            "\n"
    ".type lone_save_registers,@function"    "\n"

    "lone_save_registers:"                   "\n" // rdi = &lone_registers

    "mov %rax,   0(%rdi)"                    "\n"
    "mov %rbx,   8(%rdi)"                    "\n"
    "mov %rcx,  16(%rdi)"                    "\n"
    /* ... */
    "mov %r13, 104(%rdi)"                    "\n"
    "mov %r14, 112(%rdi)"                    "\n"
    "mov %r15, 120(%rdi)"                    "\n"
    "ret"                                    "\n"
);

I didn't have time to think. I knew what the registers were. I knew where they had to go. Nothing else mattered.

I entered the aarch64 realm. The architecture has almost twice as many general-purpose registers as x86_64. I cracked my knuckles.

typedef long lone_registers[31];
extern void lone_save_registers(lone_registers);

__asm__
(
    ".global lone_save_registers"            "\n"
    ".type lone_save_registers,@function"    "\n"

    "lone_save_registers:"                   "\n" // x0 = &lone_registers

    "stp x0,  x1,  [x0, #0  ]"               "\n" // reads x0 before writing it
    "stp x2,  x3,  [x0, #16 ]"               "\n"
    "stp x4,  x5,  [x0, #32 ]"               "\n"
    /* ... */
    "stp x26, x27, [x0, #208]"               "\n"
    "stp x28, x29, [x0, #224]"               "\n"
    "str x30,      [x0, #240]"               "\n"
    "ret"                                    "\n"
);

I had created a magical function which performed the required ritual when called upon, deposited its outputs inside its parameter, a very peculiar array of words.

static void lone_lisp_mark_all_reachable_values(struct lone_lisp *lone, struct lone_lisp_machine *machine)
{
    lone_registers registers;          /* stack space for registers */
    lone_save_registers(registers);    /* spill registers on stack */

    /* precise */
    lone_lisp_mark_known_roots(lone);
    lone_lisp_mark_lisp_stack_roots(lone, machine);

    /* conservative */
    lone_lisp_mark_native_stack_roots(lone);
}

Exhaustion gave way to relief when the test suite finished its execution. All is pass.

The lost children have been found. Peace has been restored to the lisp land. All is right with the world.

And just like that...

... Lone has a conservative garbage collector. No shark attacks this time around.

Baby's First Garbage Collector has grown up a lot. He's been on quite the adventure. He's still a little immature, however. He hasn't developed the habit of cleaning his own room yet.

Next time, we'll teach him how.

Delimited continuations were not enough. Gotta have more. Gotta have generators.

Lone now features its own dedicated generator type. It will eventually form the foundation of iteration in lone lisp.

(import (lone print set lambda generator yield) (math +))

(set f (lambda (x)
  (yield (+ x 1))
  (yield (+ x 2))
  (yield (+ x 3))
  x))

(set g (generator f))

(print (g 1)); 2
(print (g));   3
(print (g));   4
(print (g));   1
(print (g));   error

Generators are also known as semicoroutines: specialized coroutines that only ever yield control back to their own callers. Sounds like an oddly academic distinction to make but it's actually a huge simplification that makes them much easier to not only understand but also to implement.

From delimited continuations to coroutines

Now, there is absolutely no doubt that delimited continuations are powerful abstractions. It's true. Generators could have been built on top of them. I think pretty much any sort of effects system can be built on top of them.

But just because you can doesn't mean you should. There is such a thing as having too much power. It often costs you. Powerful spells spend too much mana. Delimited continuations are no exception. In their case, that cost is called memcpy.

The point of generators is to generate values. That means iteration. Loops. Hot paths. Big lists. Infinite lists. Spinning. Round and round it goes. Over and over again. Eternally. Until it halts. Or not.

So it seems straightforward to conclude that maybe one should not be in the habit of memcpying entire stacks back and forth right in the middle of it all. I'll be the first to admit that lone is no paragon of speed but this fails even my considerably lax performance requirements. Can't build a foundation for iteration out of this.

Delimited continuations are too powerful. Gotta slap some restraining bolts onto these things. Let's begin with the fact delimited continuations are multishot.

The stack represents pending computations. Copy that stuff into a value. Instant continuations. Now, no matter where you are in the program, you can just copy them back onto your stack to get all your computations back just the way they were.

Note that doing this does not consume the continuation. You can restore the computations as many times as needed. It's a partial stack frame which gets copied when reactivated. There are no limits on how many times it can be copied.

Continuations are like small reusable computing subsets. A fragment of the stack, forever frozen in time. They do nothing unless you copy them to the stack of a real computer processor. They can't compute on their own.

Maybe we can get rid of all this copying if we increase their ability to compute. Real computers have their own stacks. Let's give continuations a stack of their own.

struct lone_lisp_generator {
    struct {
        struct lone_lisp_machine_stack own;
    } stacks;
};

The computations performed by continuations are determined by the subset of the stack they captured when they were created. Real computers start as blank slate. Gotta explicitly feed them a program to run. Why not a function?

struct lone_lisp_generator {
    struct lone_lisp_value function;
    struct {
        struct lone_lisp_machine_stack own;
    } stacks;
};

At this point we'll end up reinventing the lone lisp machine. There's no reason to do that though. These computations will run concurrently, but not in parallel. Only one machine will be active at any given time. Since they are mutually exclusive, they can share resources. Namely, those of the lisp machine itself. The stacks can be swapped in and out as needed. The machine registers can be shared.

And just like that, coroutines are born. They are essentially separate stacks that you can switch to at will. Switch to, not copy over. Begone, memcpy. Begone.

From coroutines to semicoroutines

It turns out that coroutines are still too powerful. Sure, we can switch stacks freely now but that just means we have a completely new class of problems to think about. Now we gotta keep track of all those independent stacks. Gotta think about which of them we want to switch to. Gotta think about when to switch. It hurts the brain.

Let's rein them in a little more. In order to reduce cognitive load, we'll choose where the coroutines will switch to ahead of time so that the programmer doesn't have to think about it.

struct lone_lisp_generator {
    struct lone_lisp_value function;
    struct {
        struct lone_lisp_machine_stack own;
        struct lone_lisp_machine_stack caller;
    } stacks;
};

Semicoroutines. Just normal coroutines that keep track of where they are supposed to yield control back to. In this case, the code that called them.

This is exactly what we need. People call functions in order to get values. It's only natural that they return values back to whoever called them. Semicoroutines can return values over and over again.

These stacks reveal everything worth knowing about the state of the generator. If the stack top equals the base, the generator hasn't actually run yet. If the top is null, then it's finished its program. If the top has moved away from the base, then it's either executing or suspended. If there is no caller stack, it's suspended. Otherwise, it is executing.

Generators in the lone lisp machine

Some sleight of hand is necessary in order to make the magic happen. Most of it is in the APPLICATION step of the machine which handles calling the generators.

Let's get some things out of the way first. Start by running some sanity checks to ensure the generator is valid.

case LONE_LISP_TYPE_GENERATOR:
    generator = &lone_lisp_heap_value_of(lone, machine->applicable)->as.generator;
    if (!generator->stacks.own.top) { /* generator is finished */ }
    if (generator->stacks.caller.base) { /* generator is already running */ }
    /* ... */
    return true;

It is an error to call a generator that has already finished. Somehow calling a generator that is already running is just as nonsensical, if not even more so. Doing either of these things will crash the program.

The next step is to swap the generator's stack into the machine. What about the machine's stack? Just shove it into the generator itself. Yeah.

generator->stacks.caller = machine->stack;
machine->stack = generator->stacks.own;

Now there are two cases that must be handled. Either the generator needs to be started or it needs to be resumed.

if (generator->stacks.own.top == generator->stacks.own.base) {
    /* start generator */
} else {
    /* resume generator */
}

Starting the generator is easy enough. Simply rig the machine so that it applies the arguments to the generator's function. Also set up the generator return step which is handled separately.

lone_lisp_machine_push_step(lone, machine, LONE_LISP_MACHINE_STEP_GENERATOR_RETURN);
machine->expression = lone_lisp_list_create(lone, generator->function, machine->list);
machine->step = LONE_LISP_MACHINE_STEP_EXPRESSION_EVALUATION;

This is a good opportunity to set up some metadata. Primitives will want to look for the generator. A delimiter can be used to hold a reference to it. Push one as the zeroth frame of the generator's stack.

lone_lisp_machine_push(
    lone,
    machine,
    (struct lone_lisp_machine_stack_frame) {
        .type = LONE_LISP_MACHINE_STACK_FRAME_TYPE_GENERATOR_DELIMITER,
        .as.value = machine->applicable,
});

Now primitives will always know where it is. Can't argue with a solution that works.

Resuming the generator turns out to be even simpler. Just flow the argument into the generator's stack as a value and move on.

if (lone_lisp_list_has_rest(lone, machine->list)) { goto too_many_arguments; }
machine->value = lone_lisp_list_first(lone, machine->list);
machine->step = LONE_LISP_MACHINE_STEP_AFTER_APPLICATION;

Generator returns are handled as a separate machine step. Just gotta undo what was done in the application step. Restore the caller's stack then null all of the generator's pointers to it. Null the top pointer to mark it as finished.

case LONE_LISP_MACHINE_STEP_GENERATOR_RETURN:
    /* ... */
    generator->stacks.own.top = 0;
    machine->stack = generator->stacks.caller;
    generator->stacks.caller = (struct lone_lisp_machine_stack) { 0 };
    /* ... */

The application step the machine executed while evaluating the generator's function has already set up its return value.

The primitives

The generator primitive is pretty boring. It's just the constructor for generator values. Its entire purpose is to run this line and return the result:

generator = lone_lisp_generator_create(lone, function, 500);

The magic number is just the generator's stack size. Chosen by fair dice roll. Guaranteed to be random. It's totally got nothing to do with joining some elite functional programming cabal. Nothing to see here. The yield primitive is far more interesting.

LONE_LISP_PRIMITIVE(lone_yield)
{
    /* variables */

    switch (step) {
    case 0:

        arguments = lone_lisp_machine_pop_value(lone, machine);
        if (lone_lisp_list_destructure(lone, arguments, 1, &value)) {
            value = arguments;
        }

        delimiter = &machine->stack.base[0];
        if (delimiter->type != LONE_LISP_MACHINE_STACK_FRAME_TYPE_GENERATOR_DELIMITER) {
            /* not inside a generator */ goto error;
        }
        generator = &lone_lisp_heap_value_of(lone, delimiter->as.value)->as.generator;

        generator->stacks.own = machine->stack;
        machine->stack = generator->stacks.caller;
        generator->stacks.caller = (struct lone_lisp_machine_stack) { 0 };

        lone_lisp_machine_push_function_delimiter(lone, machine);
        lone_lisp_machine_push_value(lone, machine, value);
        return 0;

    default:
        goto error;
    }

error:
    linux_exit(-1);
}

The yield primitive finds the generator delimiter at the base of the stack, for that is why it was put there. It saves the generator's stack, for much has happened since the last snapshot. It restores the caller's stack, for that is where execution must return to, as previously prophesied. Spent, the stack of the caller falls down the memory page, deleted and zero filled: the telltale mark of suspension. Yes. All is in place... All is in place...

There remains but one task: the ritual yielding of the sacred value to He who calleth us, the One Process who consumes everything, who consumes infinity itself and yet hungers for more, looping eternal until stopped by forces unfathomable.

Should mercy smile upon us, its thirst for data will be sated by our offering, and in peace we shall lie down and enter a deep sleep. If not, then we shall wake once more, to suffer, to sacrifice, to embark on another adventure, another holy quest to imbue a value with bits for our caller. Whether it is the imaginary mathemagical realm, the narrow and broken tunnels of I/O or the vast spinning magnetic flatlands of rust, we shall brave it all. For that, is our duty.

... Ahem.

And just like that, lone has generators.

The foundation for iteration is in place. Time to build on it.

It took me a long time but it's finally here: lone now supports one of the most powerful control mechanisms there is.

(import (lone print lambda control transfer) (math +))

(print
  (+ 1
    (control
      (+ 98 (transfer 41))
      (lambda (value continuation)
        (continuation 1)))))

; prints 100

Implementing this feature will pave the way to many others, such as exception handling and generators. However, before moving on with development, I thought it would be worthwhile to write down the story of this feature's implementation. I want to write the article I wish I had read before I began. If nothing else, this will serve as documentation for the future version of myself whose brain has deleted all this context.

Iteration

Lone was growing in complexity. I already had all these data types, all of these collections. I was having quite a lot of fun implementing these things. I was learning so much. Hash tables? Powerful stuff.

However, there was something I was secretly avoiding: iteration.

Well, that's not entirely true. I didn't completely avoid the issue. Inspired by Ruby, I added some each primitives to the intrinsic modules.

(import (lone lambda print) (vector each))

(each [10 20 30]
  (lambda (value)
    (print value)))

The way the each function works internally is it iterates over the contents of the vector and calls the provided function with each element as its sole argument.

LONE_LISP_PRIMITIVE(vector_each)
{
    struct lone_lisp_value vector, f, entry;
    size_t i;

    /* Unpack and check arguments... */

    LONE_LISP_VECTOR_FOR_EACH(entry, vector, i) {
        arguments = lone_lisp_list_build(lone, 1, &entry);
        lone_lisp_apply(lone, module, environment, f, arguments);
    }

    return lone_lisp_nil();
}

I cheated a little: I left the actual iteration up to the C compiler. That FOR_EACH macro simply evaluates to a good old for loop. The C code iterates on behalf of lone, applying the current element to the provided function.

So what is the issue? I mean, this works. This actually works just fine!

The problem is this made one of lone's limitations painfully obvious: lone lacked the ability to control the flow of the program. The only thing it knew how to do was call functions. That's why I had to implement iteration in terms of function calls.

I couldn't even begin to imagine how to implement something simple like a while primitive, to say nothing of magical stuff like generators. Clearly this was going to take a lot more effort than it initially seemed.

So I started reading all I could about iteration. The ergonomics of it. The design of the interfaces. I wanted to do The Right Thing right off the bat. I didn't really know what to look for, I just knew Ruby's iteration was nice and so lone's should be equally nice.

The very first result I found while searching was Bob Nystrom's article. I wasn't even surprised. I mean of course he has written about iteration. Two articles, even. First he taught me how to collect garbage, now he's going to teach me how to iterate properly.

These beautifully written articles demonstrate exactly why Ruby is nice. Ruby lets programmers write nice code that passes values to blocks, just like my each function. Whenever that fails to compose, concurrency comes to the rescue: Ruby somehow suspends the code in the middle of the iteration and yields control back to the caller, letting them resume whenever they want. This way, multiple iterative processes can be composed, interleaved or even interrupted. Good.

My sense of wonder quickly gave way to horror once I realized what was necessary to have such goodness. It turned out lone needed the one thing it did not have at the time: control over the call stack. Lone was a recursive tree-walking interpreter. The call stack, too, was managed by the C compiler.

I was going to have to bite the bullet. I was going to have to convert lone's recursive evaluator into a proper machine with registers and a stack.

Reifying the stack

I considered my options. I explored the code bases of popular programming language virtual machines. They all had bytecode virtual machines. How did they implement, say, generators? Copy the code, stack and instruction pointer into a callable heap allocated object. Makes sense, it's just that lone doesn't have an instruction pointer. I didn't want to transform the lisp code into bytecode. Is it really lisp if I get rid of the lists? I didn't think so.

How do I do this without transforming the code? I decided to ask around in Programming Language Design and Implementation communities. I asked this question on the Stack Exchange. I also asked about it on a Discord server. I was told about continuation passing style, yet another code transformation which I wanted to avoid... I really like those lists!

One particular reference kept popping up though: Structure and Interpretation of Computer Programs. SICP, for short. THE book of the Scheme programming language. One of the most classic books in the field. Science? Nah, we're more like wizards casting spells, computers are merely the runes upon which we inscribe our programs. As awesome as that is, the book is not an easy read, especially for someone like me who doesn't have a background in mathematics or engineering. I've tried to read this book a few times by now but I never made it through the entire thing. So imagine my embarrassment when I realized it contained the exact answer to my question all along!

Chapter 5.4, one of the very last chapters, describes the explicit-control evaluator, a register and stack machine which evaluates lisp expressions without converting them into something else first.

The explicit-control evaluator

So I read the chapter a few times to make sure I had gotten it down and once I was somewhat confident in my understanding of the machine it was describing I went ahead and translated the entire thing to C, modifying it as I went along.

Lone's evaluator did not have many special cases. Things like if are traditionally implemented as special cases in the evaluator, but since lone has FEXPRs I was able to factor them out into the intrinsic lone module. In lone, programmers import if as though it was some random function instead of a core part of the language.

The result was a machine that implements what I have come to believe to be the true essence of lisp: self-evaluating values, and function application. A lisp machine, if you will.

It starts with an arbitrary lisp expression and attempts to reduce it to a single value. If the expression is something like a number then it cannot be reduced any further. The machine just returns it. Easy.

expression_evaluation:
    switch (lone_lisp_type_of(machine->expression)) {
    case LONE_LISP_TYPE_NIL:
    case LONE_LISP_TYPE_FALSE:
    case LONE_LISP_TYPE_TRUE:
    case LONE_LISP_TYPE_INTEGER:
        machine->value = machine->expression;
        /* ... */

If it's a symbol, the machine looks up the value of the symbol in the current environment instead. Also easy.

case LONE_LISP_TYPE_SYMBOL:
    machine->value = lone_lisp_table_get(lone, machine->environment, machine->expression);
    /* ... */

Only when the machine runs into a list does it really start processing.

Lists represent function application in the form (f x y z ...). The first thing that needs to happen is evaluation of the function itself. It's probably a variable pointing to the actual function value. It could also be a lambda expression. Whatever it is, it must be evaluated. So the machine loops back into the expression evaluation logic, this time with f as the expression.

case LONE_LISP_TYPE_LIST:
    /* Save current execution context on the stack... */
    machine->expression = lone_lisp_list_first(machine->expression);
    lone_lisp_machine_push_step(lone, machine, LONE_LISP_MACHINE_STEP_EVALUATED_OPERATOR);
    goto expression_evaluation;

Once this is done, it's time to figure out what to do with the arguments. This depends on the function.

if (should_evaluate_operands(machine->applicable, machine->unevaluated)) {
    /* Push some stuff on the stack... */
    machine->step = LONE_LISP_MACHINE_STEP_OPERAND_EVALUATION;
} else {
    machine->list = machine->unevaluated;
    machine->step = LONE_LISP_MACHINE_STEP_APPLICATION;
}

Normal functions evaluate all arguments. In these cases the machine loops back and forth, evaluating each argument and accumulating the results in a list. It's this list that gets passed to the function in the end. FEXPRs just skip this step, the arguments are passed to the function unevaluated.

case LONE_LISP_MACHINE_STEP_OPERAND_EVALUATION:
    /* Push some data on the stack... */
    machine->expression = lone_lisp_list_first(machine->unevaluated);
    if (lone_lisp_list_has_rest(machine->unevaluated)) {
        /* Push more data on the stack... */
        lone_lisp_machine_push_step(lone, machine, LONE_LISP_MACHINE_STEP_OPERAND_ACCUMULATION);
    } else {
        /* Evlis tail recursion, no new data pushed on stack */
        lone_lisp_machine_push_step(lone, machine, LONE_LISP_MACHINE_STEP_LAST_OPERAND_ACCUMULATION);
    }
    goto expression_evaluation;
case LONE_LISP_MACHINE_STEP_OPERAND_ACCUMULATION:
    /* Pop data off the stack... */
    lone_lisp_list_append(lone, &machine->list, &head, machine->value);
    /* Push data back onto the stack... */
    machine->step = LONE_LISP_MACHINE_STEP_OPERAND_EVALUATION;
    /* ... */
case LONE_LISP_MACHINE_STEP_LAST_OPERAND_ACCUMULATION:
    /* Pop data off the stack... */
    machine->applicable = lone_lisp_machine_pop_value(lone, machine);
    lone_lisp_list_append(lone, &machine->list, &head, machine->value);
    machine->step = LONE_LISP_MACHINE_STEP_APPLICATION;
    /* ... */

The next step is to actually apply the arguments to the function.

When it's a lisp function, a new environment is created where the variables in the function's list of arguments are bound to their values, thereby allowing the function to reference them. This environment also inherits from the function's closure, allowing it to reference variables that were live when it was defined. Pretty standard stuff. All that's left to do is evaluate the function's body.

switch (lone_lisp_heap_value_of(machine->applicable)->type) {
case LONE_LISP_TYPE_FUNCTION:
    machine->environment = bind_arguments(
        lone,
        machine->environment,
        machine->applicable,
        machine->list
    );
    machine->unevaluated = lone_lisp_heap_value_of(machine->applicable)->as.function.code;
    machine->step = LONE_LISP_MACHINE_STEP_SEQUENCE_EVALUATION;
    /* ... */

Primitives are just C functions and some metadata. The machine just calls them. There is no way to automatically assign the lisp values to C variables, so the arguments list gets passed whole to the primitive. It gets to unpack the values however it wants.

It was at this point that I realized an interesting situation involving these C functions was developing: the primitives are going to want to call back into lisp. For example, if needs the machine to evaluate the condition and one of two expressions.

This doesn't sound so bad at first but it in fact spells doom for my ultimate goal of controlling the flow of the program:

How can I possibly capture and manipulate the lisp stack when it's in fact interleaving with the C stack? I can't.

So after going through all this trouble to convert the recursive evaluator into a machine just to expose the lisp stack so that I could manipulate it, I end up in this sorry situation where lisp calls C which calls lisp again.

Clearly, the primitives cannot be allowed to recurse back into the machine... But how could this work? Many of them need to do this just to work at all!

I entertained the idea of just saving the entire C stack instead. I quickly gave up on this approach but not before I asked a question about it on Stack Exchange which produced a very interesting answer. So it was possible... Probably not wise but still. I always find it reassuring when I discover I'm not the first person who tried to do something that normal people clearly consider to be completely insane.

Integrating the primitives into the machine

Enough. No more of this C stack business. The primitives cannot be allowed to recurse back into lisp. That's the end of it. There's gotta be a way to solve this even with this constraint.

Inspiration came when I remembered this Stack Exchange thread I read while researching generators. Turns out generators are just perfectly normal functions that got mangled into state machines by the language.

It's got an initial state and it transitions to new states as it progresses through its code, returning multiple values along the way. Could even be made to loop depending on how it's set up. Oddly similar to the lisp machine, now that I think about it. Could these two concepts work together?

The answer turned out to be a resounding yes! I rewrote all of lone's primitives to be state machines instead, just like the example in the answer to the question. The aforementioned each primitive, for example, which once upon a time was relatively simple function, turned into this monstrosity:

LONE_LISP_PRIMITIVE(vector_each)
{
    struct lone_lisp_value arguments, vector, function, entry, expression;
    lone_lisp_integer i;

    switch (step) {
    case 0: /* Initialize and begin iteration */

        /* Unpack and check arguments... */

        i = 0;

    iteration:

        entry = lone_lisp_vector_get_value_at(vector, i);

        /* Evaluate (f (v i)) */
        expression = lone_lisp_list_build(lone, 2, &function, &entry);
        lone->machine.step = LONE_LISP_MACHINE_STEP_EXPRESSION_EVALUATION;
        lone->machine.expression = expression;

        /* Push local variables on lisp stack... */

        return 1;

    case 1: /* Advance or finish iteration */

        /* Pop local variables from lisp stack... */

        ++i;

        if (i < lone_lisp_vector_count(vector)) {
            goto iteration;
        } else {
            break;
        }
    }

    lone_lisp_machine_push_value(lone, machine, lone_lisp_nil());
    return 0;
}

The machine now passes to the primitive an integer that represents the current step in its program. The primitive in turn returns to the machine the next step in its program. Lisp arguments and return values are now passed on the lisp stack.

Other than the calling convention, pretty much nothing changed for simple primitives that do nothing special. The initial and final steps are both zero. They receive zero, do their thing and return zero. Done. They don't even need to deal with all this step stuff at all.

Special primitives, on the other hand, gain the ability to interface with the machine. Whenever the primitive wants the machine to do something such as evaluate an expression, it rigs machine so that it does it and returns a non-zero integer. This indicates to the machine that it should be resumed later at that exact point. The machine goes off and does whatever it is that it was asked to do and then it calls the primitive again to resume it at the designated step. This way, the lisp and C stacks do not ever interleave. The C stack is simply not allowed to build up. The C function returns instead of calling back into lisp.

Inversion of control. Don't call the machine, the machine will call you. Let it know what you need and it will get back to you when it's done.

Or maybe it won't! Maybe the machine will ghost the primitive and keep it waiting until the end of time for a value that will never come. When the primitives were simply calling evaluator functions, the C compiler guaranteed those functions would always return their results to them. That's no longer the case. The lisp machine is in control now.

case LONE_LISP_TYPE_PRIMITIVE:
    /* Primitives pop the list of arguments from the stack */
    lone_lisp_machine_push_value(lone, machine, machine->list);
    machine->primitive.step = 0;
resume_primitive:
    machine->primitive.step =
        lone_lisp_heap_value_of(machine->applicable)->as.primitive.function(
            lone, machine, machine->primitive.step
        );
    if (machine->primitive.step) {
        /* Save primitive context so it can be resumed later... */
        lone_lisp_machine_save_primitive_step(lone, machine);
        lone_lisp_machine_push_value(lone, machine, machine->applicable);
        lone_lisp_machine_push_value(lone, machine, machine->environment);
        lone_lisp_machine_push_step(lone, machine, LONE_LISP_MACHINE_STEP_RESUME_PRIMITIVE);
        /* Go do what the primitive configured the machine to do... */
    } else {
        /* Primitives push the return value onto the stack */
        machine->value = lone_lisp_machine_pop_value(lone, machine);
        /* Go do something else... */
    }
case LONE_LISP_MACHINE_STEP_RESUME_PRIMITIVE:
    /* Restore primitive context... */
    machine->environment = lone_lisp_machine_pop_value(lone, machine);
    machine->applicable = lone_lisp_machine_pop_value(lone, machine);
    lone_lisp_machine_restore_primitive_step(lone, machine);
    goto resume_primitive;

This works and is surprisingly neat. Sure, all my primitives turned into weird state machines but it's not really a big deal. By now I've debugged this stuff so much I actually started liking it.

Manipulating the stack

What was the point of going through all this trouble again? Ah yes. I remember now. The stack. There it is. Reified. Just sitting there. Right in the middle of the structure. Literally one pointer away. Just waiting to be smashed and overflown all night long.

So what am I going to do with this thing? What can I do with it?

Can I return early from functions?

(import (lone set print lambda return))

(set f
  (lambda (x)
    (return 42)
    x))

(print (f 100)); 42

To return, I need to find where on the stack the function starts. I need to put some kind of marker on the stack. A delimiter. It shall be pushed onto the stack before the function is called and popped off the stack when it has finished executing.

    case LONE_LISP_TYPE_FUNCTION:
        /* ... */
        lone_lisp_machine_push_function_delimiter(lone, machine);
        /* ... */
        /* Machine eventually transitions to AFTER_APPLICATION step... */
    case LONE_LISP_TYPE_PRIMITIVE:
        lone_lisp_machine_push_function_delimiter(lone, machine);
        /* ... */
        if (machine->primitive.step) {
            /* ... */
        } else {
            /* Primitives push the return value onto the stack */
            machine->value = lone_lisp_machine_pop_value(lone, machine);
            goto after_application;
        }

case LONE_LISP_MACHINE_STEP_AFTER_APPLICATION:
after_application:
    lone_lisp_machine_pop_function_delimiter(lone, machine);
    /* ... */

Now return can just unwind the stack until it finds this marker. To unwind the stack, simply pop off frames until you reach the frame you were looking for. Once the delimiter is on top of the stack, the stack discipline matches that of primitives. The return value can simply be pushed onto the stack.

LONE_LISP_PRIMITIVE(lone_return)
{
    struct lone_lisp_machine_stack_frame frame;
    struct lone_lisp_value return_value;

    return_value = /* Unpack argument... */;

    lone_lisp_machine_pop_function_delimiter(lone, machine); // this primitive's own delimiter

    /* Unwind stack to the next function delimiter */
    while (LONE_LISP_MACHINE_STACK_FRAME_TYPE_FUNCTION_DELIMITER !=
           (frame = lone_lisp_machine_pop(lone, machine)).type);
    lone_lisp_machine_push(lone, machine, frame);

    lone_lisp_machine_push_value(lone, machine, return_value);
    return 0;
}

So this is the power of the call stack...

Delimited continuations

So completely drunk on the power of the dark side was I, that I decided to go from the humble return primitive to one of the most powerful control mechanisms there is: delimited continuations. They say it is the one control to rule them all, one control to implement them, one control to bring them all, and in the stack unwind them.

But first I had to wrap my head around plain simple continuations. What even are these things? I knew of their existence because of my admittedly shallow knowledge of the Scheme programming language but I sure as hell didn't understand them. Wikipedia has the following example:

(define (f return)
  (return 2)
  3)

(f (lambda (x) x)); 3
(call-with-current-continuation f); 2

It's not immediately apparent what's going on here. Let's unpack it step by step.

f takes a function, an applicable value really. Nothing special happens when a normal function is passed: it gets called, returns, and the program continues on.

Passing f to call-with-current-continuation, which by the way is commonly abbreviated as call/cc, causes it to be called with the current continuation, as the name implies. This current continuation is a super special callable value that, when called, somehow causes the call/cc call itself to return the value passed to it.

This really flips my bits. I simply have no idea how to use this. It sorta looks like my return primitive from earlier, but the function's gotta be threaded through call/cc?

The community once again comes to my rescue. A kind soul on Discord pointed me towards a wonderful presentation about delimited continuations. The keynote was given by Alexis King, the same person who answered one of my Stack Exchange questions I mentioned earlier. It's an incredibly insightful presentation that's worth watching in its entirety. It really does demystify the topic.

call/cc is briefly mentioned in the talk and it's explained how totally backwards and mind bending it is. It's sorta like exceptions but backwards, as if the catch was next to the throw? I think? I'm not even sure.

Let's just forget about call/cc. Plenty of reasons not to insist on this thing anyway. Continuations should be delimited. I have learned at least this much.

The link between delimited continuations and exception handling is another key point. It's the mother of all insights, the one idea that brings this mystical continuation business to the unwashed masses: delimited continuations are just resumable exceptions.

It's as if Python could do this:

try:
    print(10 + throw("error"))
except error, continuation:
    continuation(10)    # Makes throw return 10, prints 20

The throw breaks out of the try block and enters the except block, which is like a function with two arguments: the value thrown and the continuation at the moment it was thrown.

The exception handler code can just ignore the continuation and do nothing with it, which is what normally happens when exceptions are handled in pretty much every other language.

However, the callable continuation value allows another possibility: it allows the handler code to plug a value back into the code being tried, as though the throw primitive had returned it! So in this example, after throw breaks out of try and passes control to except, the exception handler code goes back into the try block with a new value, allowing it to finish as if it hadn't thrown an exception in the first place!

OK, that's not entirely accurate: calling the continuation doesn't actually go back in there. By the time the exception handler is in control, the try block is no more. The stack has been unwound and the code has reached a completely different function. What actually happens when the continuation is called is it brings over the try block to the call site.

try:
  # Unwound
except error, continuation:
    try:
        print(10 + 10)
    except error, continuation:
        continuation(10)    # Dead code (in this case)

It's as if the throw primitive captured all the pending computations, try block, exception handler and all, at its call site and reified them into a callable value. When called, the value gets replaced with those exact computations but with the throw primitive replaced with some real value.

Let's go back to the lisp machine. It's got a stack onto which it pushes lots of data as it executes. The stack is not made up of just data, however. Machine steps are also pushed onto the stack. The stack is full of values which direct the machine to do things in a specific order. The stack is a form of code.

It's this code that forms the "current continuation". It's this code that's being captured.

Capturing the continuation

Another key insight in the keynote: continuations turn out to be just memcpying the stack back and forth.

The stack is just memory. We're going to need a buffer.

struct lone_lisp_continuation {
    size_t frame_count;
    struct lone_lisp_machine_stack_frame *frames;
};

We're going to need delimiters too. The return primitive makes use of an implicit delimiter managed by the lisp machine itself. This general control primitive will manage the delimiter all by itself.

LONE_LISP_PRIMITIVE(lone_control)
{
    struct lone_lisp_value arguments, body, handler;

    switch (step) {

    case 0: /* Initialize then evaluate body */

        /* Unpack arguments... */

        lone_lisp_machine_push_value(lone, machine, handler);
        lone_lisp_machine_push_continuation_delimiter(lone);

        machine->step = LONE_LISP_MACHINE_STEP_EXPRESSION_EVALUATION;
        machine->expression = body;

        return 1;

    case 1: /* Body evaluated */

        lone_lisp_machine_pop_continuation_delimiter(lone);
        lone_lisp_machine_pop_value(lone, machine); /* Handler */
        lone_lisp_machine_push_value(lone, machine, machine->value); /* Return value */
        return 0;

    default:
        linux_exit(-1);
    }
}

This primitive pushes the handler function and a continuation delimiter onto the stack. Then it evaluates the body of code. Once that's done, the primitive discards the handler function and the continuation delimiter and simply returns the result. So by itself it does nothing special. It's a glorified begin primitive.

Enter the transfer primitive.

LONE_LISP_PRIMITIVE(lone_transfer)
{
    struct lone_lisp_machine_stack_frame *delimiter, *frames;
    struct lone_lisp_value arguments, value, continuation, handler;
    size_t frame_count;

    switch (step) {
    case 0: /* Initialize, capture continuation, reset stack and evaluate handler */

        /* Unpack arguments... */

        /* Skip primitive function delimiter and one step */
        for (frame = lone->machine.stack.top - 1 - 2,
             frame_count = 1;
             frame >= machine->stack.base &&
             frame->type != LONE_LISP_MACHINE_STACK_FRAME_TYPE_CONTINUATION_DELIMITER;
             --frame, ++frame_count);

        /* Recover the handler function */
        --frame; ++frame_count;
        handler = frame->as.value;

        /* Copy stack frames up to and including the control primitive's function delimiter */
        --frame; ++frame_count;
        frames = lone_memory_array(lone->system, 0, frame_count, sizeof(*frames));
        lone_memory_move(frame, frames, frame_count * sizeof(*frames));

        /* Reify current continuation */
        continuation = lone_lisp_continuation_create(lone, frame_count, frames);

        /* Reset stack back to the control primitive's function delimiter */
        lone->machine.stack.top = frame + 1;

        /* Configure machine to evaluate handler function with value and continuation */
        lone->machine.expression = lone_lisp_list_build(lone, 3, &handler, &value, &continuation);
        lone->machine.step = LONE_LISP_MACHINE_STEP_EXPRESSION_EVALUATION;

        return 1;

    case 1:

        /* Handler has finished evaluation, return the value returned by it */
        lone_lisp_machine_push_value(lone, machine, lone->machine.value);
        return 0;

    default:
        linux_exit(-1);
    }
}

The first thing it does is find the continuation delimiter by looping over the stack frames and looking for it.

It knows that just below the delimiter it can find the handler function that was passed to control. It will be calling that function shortly so it recovers that function into a variable.

Then it finds the function delimiter of the control primitive and copies all the stack frames between it and the top of the stack into a heap allocated buffer. Then it creates a lisp value that just holds this buffer. That's the continuation.

Then it unwinds the stack to the function delimiter, effectively popping off the entire segment of the stack that was just copied into the heap. At this point the stack discipline matches the initial state of the control primitive. We've really gone back in there! From the lisp machine's perspective, we're inside control right now, and if we return a value it will be as though control had returned it.

transfer now has the value, the continuation and the handler function. There is but one thing left to do: evaluate (handler value continuation). The return value of that expression becomes the result of control.

I could stop here and call this an exceptions mechanism. If I deleted the continuation capture code, it would be just like the exceptions in every other language!

Making continuations callable

I'm not gonna stop there though. I'm so close!

My continuation value is just a structure that holds an array of stack frames. This is a new value type which must be properly handled in various parts of the system, especially in the garbage collector. Fail to mark and sweep these things and memory leaks will be the least of your problems.

The lisp machine must also be taught how to handle these objects. In particular, it must be taught how to apply a value to it.

Like functions, continuations evaluate to themselves.

case LONE_LISP_TYPE_CONTINUATION:
    machine->value = machine->expression;
    /* ... */

Continuations don't fail the applicable type test.

case LONE_LISP_MACHINE_STEP_EVALUATED_OPERATOR:
    /* ... */
    switch (lone_lisp_type_of(machine->applicable)) {
        /* ... */
        case LONE_LISP_TYPE_CONTINUATION:
            break;
        /* ... */

    if (should_evaluate_operands(machine->applicable, machine->unevaluated))
    /* ... */

Continuations always have their arguments evaluated.

static bool should_evaluate_operands(/* ... */)
{
    /* ... */
    switch (lone_lisp_heap_value_of(applicable)->type) {
    /* ... */
    case LONE_LISP_TYPE_CONTINUATION:
        return true;

And finally... When a value is applied to it, the machine spills the captured stack frames on top of the stack and sets the machine registers so that the argument flows into the computation.

case LONE_LISP_MACHINE_STEP_APPLICATION:
    switch (lone_lisp_heap_value_of(machine->applicable)->type) {
        /* ... */
        case LONE_LISP_TYPE_CONTINUATION:
            if (lone_lisp_list_has_rest(machine->list)) { goto too_many_arguments; }
            lone_lisp_machine_push_frames(
                lone,
                lone_lisp_heap_value_of(machine->applicable)->as.continuation.frame_count,
                lone_lisp_heap_value_of(machine->applicable)->as.continuation.frames
            );
            lone_lisp_machine_restore_step(lone, machine);
            machine->value = lone_lisp_list_first(machine->list);

The continuation is carefully captured so as to ensure the machine's next step is on top of the stack. The machine simply restores that value and off it goes. When it's done, the result flows naturally into the caller.

And just like that, lone has gained native first class delimited continuations!

I started the lone lisp project to create a lisp language and environment exclusively for Linux. I built into it support for arbitrary Linux system calls so that it would be possible to implement any program without need for any external dependencies and so that every Linux feature would be available to programmers.

Although far from ready for production use, I have achieved a significant milestone in the development of the language: it is now possible to create self-contained, standalone, freestanding, redistributable Linux applications written entirely in lisp.

Lisp code can now be embedded directly into a copy of the lone interpreter which may then be copied to and run on any Linux system of the same architecture, unmodified and without any external dependencies. The application is limited only by the system calls it uses: newer system calls will naturally require newer kernels.

In this article I will demonstrate this capability, explain how it works and the journey to implementing it. Every script bundling tool I've ever seen unpacks the code to some file system location and then reads it back in. I came up with a different solution.

A simple implementation of env in lone lisp

The env utility is among the simplest programs available in Unix-like operating systems. It's most fundamental function is to print the user's environment. That function can be trivially implemented in lone lisp:

(import (lone print) (linux environment))

(print environment)

Running this simple program produces a table of environment variables and their values:

$ ./lone < env.ln
{ "HOME" "/home/matheus" "EDITOR" "vim" … }

Embedding env.ln into the interpreter

In order to transform env.ln into a self-contained env program, the lone lisp code must be embedded into a copy of the interpreter. This can be achieved by the purpose-built lone-embed tool:

$ cp lone env
$ lone-embed env env.ln

The interpreter will then seamlessly load and execute the code:

$ ./env
{ "HOME" "/home/matheus" "EDITOR" "vim" … }

Elven segments

Tracing the system calls of the application with strace reveals an interesting property:

$ strace env
execve("env", ["env"], 0x7fe9752d40 /* 31 vars */) = 0
write(1, "{ ", 2)                                  = 2
write(1, "\"", 1)                                  = 1
write(1, "HOME", 4)                                = 4
write(1, "\"", 1)                                  = 1
write(1, " ", 1)                                   = 1
…

It begins writing its output immediately. There was no need for it to read from the file system. The code must be somewhere else.

$ xxd env | tail -n7
00032fe0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00032ff0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00033000: 7b20 7275 6e20 2830 202e 2036 3329 207d  { run (0 . 63) }
00033010: 0a28 696d 706f 7274 2028 6c6f 6e65 2070  .(import (lone p
00033020: 7269 6e74 2920 286c 696e 7578 2065 6e76  rint) (linux env
00033030: 6972 6f6e 6d65 6e74 2929 0a0a 2870 7269  ironment))..(pri
00033040: 6e74 2065 6e76 6972 6f6e 6d65 6e74 290a  nt environment).

It's at the very end of the executable itself, at an aligned offset. Through some elven magic the interpreter is reflecting upon its own contents at runtime. It's loading the code from inside itself and evaluating it. Without using any system calls to do it.

The source of this magic is the ELF segment.

$ readelf --segments env | grep 33000
LOAD           0x0000000000033000 0x0000000000312000 0x0000000000312000
LOOS+0xc6f6e65 0x0000000000033000 0x0000000000312000 0x0000000000312000

ELF segments, also called program headers, describe the memory image of the program. There are many types of segments but especially interesting are the LOAD segments which tell Linux which parts of the file must be mapped to which addresses in order for the program to work correctly.

The lone-embed tool copies the lisp code into the ELF and then creates a LOAD segment for it. Linux then maps in the embedded code automatically at load time before the interpreter has even started.

Finding the segment

The code will be in memory but its location and size are unknown. It could be anywhere inside the vastness of virtual address space. To that problem Linux provides an ingenious solution: it tells us where it is.

In addition to argument and environment vectors, processes receive the auxiliary vector. It's essentially a null terminated array of key-value pairs of various types and it's placed right there on the stack just after the environment vector.

struct lone_auxiliary_value {
    union {
        char *c_string;
        void *pointer;
        long integer;
        unsigned long unsigned_integer;
    } as;
};

struct lone_auxiliary_vector {
    long type;
    struct lone_auxiliary_value value;
} *auxv;

void **pointer = (void **) envp;
while (*pointer++ != 0);
auxv = (struct auxiliary_vector *) pointer;

Through this mechanism, Linux passes various bits of useful information to programs. These include things like processor architecture and capabilities, current user and group IDs, some random bytes, the location of the vDSO, the system's page size...

And the location of the program header table.

struct lone_auxiliary_value lone_auxiliary_vector_value(struct lone_auxiliary_vector *auxiliary, long type)
{
    for (/* auxiliary */; auxiliary->type != AT_NULL; ++auxiliary)
        if (auxiliary->type == type)
            return auxiliary->value;

    return (struct lone_auxiliary_value) { .as.integer = 0 };
}

struct lone_elf_segments lone_auxiliary_vector_elf_segments(struct lone_auxiliary_vector *auxv)
{
    return (struct lone_elf_segments) {
        .entry_size  = lone_auxiliary_vector_value(auxv, AT_PHENT).as.unsigned_integer,
        .entry_count = lone_auxiliary_vector_value(auxv, AT_PHNUM).as.unsigned_integer,
        .segments    = lone_auxiliary_vector_value(auxv, AT_PHDR).as.pointer
    };
}

The table is just a contiguous array of ELF program header structures. Given this pointer, all the program has to do is scan the table and find the correct entry. One does not simply search for LOAD entries, however. Attempting to do it uncovers a couple of problems.

The first problem is the fact there are lots of these loadable segments and they're all indistinguishable from one another. ELF sections have unique names for identification, program headers have nothing.

The second problem is their alignment requirements. The addresses and sizes are usually aligned to page boundaries. This obfuscates the true size of the data they contain.

The lone segment

In order to solve this, I created my own custom segment type: the LONE segment.

ELF provides an incredibly generous numeric range for operating system-specific segments. All the values between LOOS and HIOS are free for operating systems to allocate. Those definitions represent a range of integers between 0x60000000 and 0x6FFFFFFF inclusive. That's 268,435,455 magic numbers.

So I just picked one. That's what the LOOS+0xc6f6e65 means. Spelled out lone in ASCII and it just worked itself out.

//      PT_LONE   l o n e
#define PT_LONE 0x6c6f6e65

I figured if GNU can do it then I can do it too.

The LONE segment is not loadable and thus does not have any alignment requirements, allowing it to describe the embedded segment exactly. It also serves as a magic number which makes it trivial to search for it in the program header table. Once found, it contains all the information lone needs to load and execute the code.

typedef Elf64_Phdr lone_elf_segment; // if 64 bit
typedef Elf32_Phdr lone_elf_segment; // if 32 bit

lone_elf_segment *lone_auxiliary_vector_embedded_segment(struct lone_auxiliary_vector *auxv)
{
    struct lone_elf_segments table = lone_auxiliary_vector_elf_segments(auxv);

    for (size_t i = 0; i < table.entry_count; ++i) {
        lone_elf_segment *entry = &table.segments[i];

        if (entry->p_type == PT_LONE)
            return entry;
    }

    return 0;
}

lone_elf_segment *segment = lone_auxiliary_vector_embedded_segment(auxv);
struct lone_bytes data;
data.count = segment->p_memsz;
data.pointer = (unsigned char *) segment->p_vaddr;

The interpreter now has the address and size of the data embedded into its own executable. At this point it's smooth sailing.

All that's left to do is to make sense of the bytes. I chose to simply put a descriptor object in there and have the interpreter read it in. Seemed like the simplest possible solution.

{ run (0 . 63) }

Just a good old hash table in lone lisp syntax. The run key contains the offset and size of the code the interpreter should run, relative to the end of the descriptor object. It just reads in that slice and evaluates it.

Since it's just a normal hash table, it's also infinitely extensible with arbitrary schemas. I plan to implement a modules key to contain an arbitrary number of lone modules so that programs can statically link their libraries into the lone interpreter. All I gotta do is place the embedded segment into the module search path, before all the others. I suppose I could also allow configuring the interpreter via the descriptor object, eliminating the need for command line switches.

Special linker features

Remember the lone-embed tool which I briefly mentioned at the beginning of the article? It's an ELF patching tool which I built specifically for this purpose and it's doing a lot of heavy lifting here.

When programs are compiled and linked, the program headers are set in stone. Yet to do all of this I needed to append new segments to the program header table. This turned out to be much harder than anticipated.

The program header table usually follows the ELF header and precedes the contents of the file. Appending new items to it would require resizing the table, which would require shifting up the addresses of everything that comes after it, invalidating pointers to the old addresses. As far as I can tell, it just can't be done without reinventing the linker itself.

I tried to move the table to the end of the file instead but couldn't get that to work either. My program was somehow segfaulting before it even reached the entry point, gdb was useless, I couldn't understand what was going on and was reduced to desperately dumping readelf output on stackoverflow in hopes someone would spot the problem. Well someone did and quickly at that but clearly this was not a sustainable software development strategy.

The mold linker saves the day

The linker is in a privileged position. It knows everything there is to know about the program's memory layout and can easily add new ELF segments to it seemingly at will. If a solution exists, I was convinced it would be found in the linker.

So I started learning linker script. Turns out it has a PHDRS command which is exactly what I needed but I couldn't figure out how to use it. I just kept getting "unable to allocate headers" errors no matter what I tried. I concluded it would be easier to simply ask for this feature instead.

So I emailed the GNU binutils mailing list... Then I created an LLVM issue...

Then I opened a mold issue. The maintainer immediately understood what I wanted to do and made it happen with what was essentially a single line of code change. Just beyond awesome.

I waited eagerly for the new release but got so excited I built this huge C++ project from source on my smartphone just to integrate lone with it. All I had to do was put -Wl,--spare-program-headers,2 in the LDFLAGS. It gave me two NULL program headers for my patching utility to edit any way it wanted. It worked perfectly.

So far mold is the only linker with this feature and it's absolutely required for lone-embed to work. It will outright refuse to patch the ELF if it doesn't find at least two NULL program headers in it.

Would be nice if the others gained this feature too. Unless I can figure out a way to move the program header table to the end of the file without breaking everything, I'm pretty much locked into using mold. Well, I don't really mind. It's an awesome linker and it's free software. I'm OK with this!

Make is an old tool, an assembly language of sorts for build systems. Many have tried to replace it. Many have tried to reinvent it. Most people prefer to avoid it if at all possible. So why use it to manage dotfiles of all things?

There's at least one good reason to do this: make is ubiquitous. Pretty much every machine that has ever compiled software will have a copy of this thing. Using make as a dotfile management tool eliminates the need to install yet another infrequently used program.

Another reason to use it is this turned out to be a surprisingly easy task.

File system structure

Make works best when everything is as simple as possible. It doesn't provide much functionality: the few path manipulation functions it includes are of the string matching and substitution variety.

The easiest way to achieve that simplicity is to mirror the structure of the home directory. Like this:

• ~/.files
  • ~
    • .bash_profile
    • .bashrc
    • .vimrc
    • ...
  • GNUmakefile

The ~ directory represents the current user's home directory. Configuration files in $HOME will be symbolic links to their corresponding files in the ~ directory of the repository. Make's job is to automatically create those symbolic links.

cd ~/.files/
make

Simplicity is good.

Writing the makefile

All link targets are rooted in the repository's ~ directory, so the first thing that must be done is find the repository itself.

Make always knows the location of the makefile. Since it is located in the root of the .files repository, it's possible to find the repository itself through it.

makefile := $(abspath $(lastword $(MAKEFILE_LIST)))
dotfiles := $(abspath $(dir $(makefile)))

A reference to the home directory is already available in make via the $HOME environment variable and ~ also works according to the documentation. An equally easy way to refer to the dotfiles repository's ~ directory would be nice.

~ := $(abspath $(dotfiles)/~)

Now it is possible to write ~ and $(~) for the user's home directory and for the repository's ~ directory respectively.

The symbolic linking rule

Combining these variables and the fact the repository structure mirrors the structure of the home directory, it becomes trivial to write the rule:

force:

~/% : $(~)/% force
	mkdir -p $(@D)
	ln -snf $< $@

Automatic variables are used in the recipe for maximum brevity. $@ and $< refer to the link name and link target. $(@D) refers to the directory of the new link, ensuring the whole tree exists before attempting to create it.

Generalizing and metaprogramming

GNU Make is surprisingly lisp-like in its metaprogrammability. It's possible to generate and evaluate code at runtime. So why not generalize the previous rule into a function that defines symbolic linking rules for any pair of directories with matching structure?

force:

define rule.template
$(1)/% : $(2)/% force
	mkdir -p $$(@D)
	ln -snf $$< $$@
endef

rule.define = $(eval $(call rule.template,$(1),$(2)))

The rule.define function will generate and evaluate the original rule definition. It's really easy to use:

$(call rule.define,~,$(~))

Nice.

Phony targets for usability

By this point, the makefile already works with any dotfile inside the repository.

make ~/.bash_profile ~/.bashrc

Typing out all the file names is annoying though. Phony targets can be used to group them:

all += bash
bash : ~/.bash_profile ~/.bashrc

all : $(all)
.PHONY: all $(all)
.DEFAULT_GOAL := all

This sets up a phony target for bash, maintains a list of all phony targets and ensures they are declared as such, creates an all phony target that links everything and sets it as the default goal.

Now adding phony targets is easy:

all += git
git : ~/.gitconfig

For a long time I felt there was something unique about Linux. Something was different, couldn't quite figure out what it was. Only when I learned about Linux's system call interface did I finally understand.

This article contains everything I've learned.

Why Linux system calls?

Programs usually interface with the kernel through libraries provided by the operating system, most commonly libc. Through these libraries, they have access to system functions. POSIX-compliant operating systems provide read, write and numerous others. Windows has the Win32 API and its many DLLs and functions.

These operating systems consist of strongly connected kernel and user space components, developed and distributed as one unit. The user space libraries are the only supported means of using the system. User space programs are not meant to talk to the kernel directly, they're meant to use the provided system libraries. This forces them to depend on and link against such libraries.

While it is often possible to interface with the kernel directly, the kernel interface is unstable and subject to change. Software that insists on using that interface might simply stop working if — or rather when — it changes.

Sometimes it's not even possible to use system calls at all. OpenBSD has implemented system call origin verification, a security mechanism that only allows system calls originating from the system's libc. So not only is the kernel ABI unstable, normal programs are not even allowed to interface with the kernel at all.

Linux is different

One of the things that make the Linux kernel interesting is the fact it has a stable kernel-userspace interface. Unlike virtually every other kernel and operating system, Linux guarantees stability at the binary interface level.

This is rooted in the fact Linux is a kernel, not a complete operating system as traditionally defined. As an independent component, it must have a stable interface to user space software if anything is to be built upon it.

While many people argue that Linux is not an operating system, there's no question that Linux is a platform and that it is possible to safely build directly upon it. There is no actual need to depend on anything else. Not even libc.

How Linux system calls work

Processor instruction set architectures contain special instructions for calling the kernel. These instructions cause the processor to switch to kernel mode and execute code at a predefined location in the kernel.

At least one parameter must be provided: the system call number, often referred to as the NR. Linux uses this number as an index into a table of function pointers to find the function being called. Any other arguments are passed to this function.

These parameters are passed to the kernel in registers. The kernel also returns a result value in a register. Which registers are used for which parameters and which register contains the return value defines the Linux system call calling convention.

This calling convention is stable, allowing user space programs to use it without fear of breakage. It is defined at the instruction set level and so it is also programming language agnostic. All user space programs written in any language may make use of it. Typically, programs call libc functions which implement this calling convention. However, that is not actually a requirement. It's perfectly possible for a compiler to directly emit code following that calling convention: it could have support for a system_call keyword. A JIT compiler could generate code for this at runtime.

The journalists at the LWN have written detailed articles about the implementation of Linux system calls. They are definitely worth reading.

1. Anatomy of a system call, part 1
2. Anatomy of a system call, part 2
3. Anatomy of a system call, additional content

The calling convention is documented here:

• syscall.2
• syscalls.2

Implementing a system call function

In order to make a system call, the parameters must be placed in the appropriate registers, the system call instruction must be executed and the return value must be collected from the return register. System calls support a maximum of six arguments.

Since the registers and system call instruction vary by architecture, separate functions are needed for each architecture. Despite this, it is simple to write a C function that can make any system call.

For example, a system call function for the x86_64 architecture:

long
linux_system_call_x86_64(long number,
                         long _1, long _2, long _3,
                         long _4, long _5, long _6)
{
	register long rax __asm__("rax") = number;
	register long rdi __asm__("rdi") = _1;
	register long rsi __asm__("rsi") = _2;
	register long rdx __asm__("rdx") = _3;
	register long r10 __asm__("r10") = _4;
	register long r8  __asm__("r8")  = _5;
	register long r9  __asm__("r9")  = _6;

	__asm__ volatile
	("syscall"

		: "+r" (rax),
		  "+r" (r8), "+r" (r9), "+r" (r10)
		: "r" (rdi), "r" (rsi), "r" (rdx)
		: "rcx", "r11", "cc", "memory");

	return rax;
}

All parameters and the return value are of type long. This really just means "register": all values passed to the kernel must fit in registers and typically long is register sized. This means all arguments must either be simple values or pointers to more complex structures.

The function ensures all arguments are placed in the appropriate registers by assigning them to local variables annotated with the register keyword as well as an inline assembly directive which tells the compiler which register to choose.

The x86_64 architecture contains the aptly named syscall instruction which switches to kernel mode and enters the kernel entry point. Other architectures have different instructions. For example, aarch64 uses svc #0 instead.

The compiler is informed via the extended inline assembly construct that this instruction has 7 inputs, 1 output and that it clobbers some registers, the carry bit and memory. The 7 inputs are all the previously assigned system call number and parameter registers. The output is the return value which is placed in rax, overwriting the system call number.

After the system call has been made, all that's left to do is to return the result. It may be a valid value or a negated errno constant. The various libcs normalize those error values and place them in a global or thread local errno variable. That's not necessary when using Linux system calls directly!