Ridgun
The language itself: a compiler that compiles itself, the stable project tool, the specification and the book
| 1 | # Chapter 08: Tensors, RMM Compiled Memory (`RMM1`/`RMM2`), Regions, and Systems |
| 2 | |
| 3 | So far we have seen the friendly, elegant, mathematical side of Ridgun: pure functions, algebraic data types, *pattern matching*, decision trees with `|` / `|.`, and parallel `||` graphs with `->` that put every core of your processor to work without asking you for a single `Mutex`. |
| 4 | |
| 5 | At this point, a veteran systems programmer at the back of the room usually raises a hand with a half-smile and asks: |
| 6 | |
| 7 | > *"All of this is lovely for computing invoices or compiling ASTs, but what happens when I need to multiply matrices of millions of floats at breakneck speed? What about memory? Is there a Garbage Collector hiding under the rug waiting for the worst possible moment to freeze my program for 200 milliseconds? And how do I talk to a real Unix operating system?"* |
| 8 | |
| 9 | Excellent questions. Let's pop the hood of Ridgun and see how it handles low-level systems work, numerical computing, and deterministic memory. |
| 10 | |
| 11 | --- |
| 12 | |
| 13 | ## 8.1. The Memory Dilemma: Neither Garbage Collector Nor Borrow Checker Fights (`RMM1` and `RMM2`) |
| 14 | |
| 15 | In modern programming languages there are usually two irreconcilable camps: |
| 16 | |
| 17 | 1. **The Garbage Collector camp (Java, Go, Python, JS, Haskell, OCaml):** you allocate memory freely and, every now and then, an invisible garbage truck stops all traffic in your program to trace which objects are still alive and which are dead. It is wonderfully convenient... until you are building an ultra-fast systems tool or a strict-latency server and the garbage truck decides to double-park. |
| 18 | 2. **The manual control / Borrow Checker camp (C, C++, Rust):** there is no garbage collector, but in exchange you either juggle raw pointers by hand (risking a `Segmentation fault` or *use-after-free*) or spend half the afternoon arguing with the *borrow checker* about why `'a` does not live as long as `'b`. |
| 19 | |
| 20 | Ridgun solves this dilemma at the root with **Ridgun Memory Management (`RMM`)**, a deterministic system of two complementary policies (**`RMM1`** and **`RMM2`**) combined with **lexical regions (`with region`)**: |
| 21 | |
| 22 | ### Policy `RMM1` — Compiled Memory Management (Linear and Conditional CMM) |
| 23 | When the compiler statically knows the flow of local variables inside a function: |
| 24 | - **Linear CMM**: `ridgunc` computes at compile time the **exact last line** where each local variable (`a`, `b`, ...) is read and inserts its deterministic deallocation (`rg_cmm_drop`) immediately afterward, or transfers ownership with zero copies (`rg_cmm_move`) when the variable is returned as the result. |
| 25 | - **Conditional CMM (`ptr(a) != ptr(c)`)**: what if a function like `choose cond a b` returns either `a` or `b` depending on a runtime condition? `ridgunc` builds the ancestor table and emits a **1-CPU-cycle** pointer-identity guard: when `b` is no longer used while `c` remains alive, it checks `if ptr(b) != ptr(c) { free(b) }`. The unchosen branch is freed immediately and the chosen one survives intact. |
| 26 | |
| 27 | ### Policy `RMM2` — Assignment Counting and Structural Cascade |
| 28 | And what happens when you nest data structures inside other data structures (`xs = [a, b]`), reassign mutable variables (`mut x = ...`), or chain complex calls? That is where **`RMM2`** steps in, governing the entire lifecycle through four airtight mathematical rules: |
| 29 | |
| 30 | 1. **Incoming function arguments are untouched (`+0` on entry, `-0` on exit)**: passing a list or a record into a function never increments or decrements reference counters needlessly. |
| 31 | 2. **Every assignment to a local variable or container slot adds `+1`**: |
| 32 | - `a = [10, 20]` $\rightarrow$ block `M1` is born with counter `1`. |
| 33 | - `xs = [a, b]` $\rightarrow$ list `xs` is born with `1`, and because it holds references to `a` and `b`, both climb to `2`. |
| 34 | - If you reassign a mutable variable (`mut x = ...` and later `x = new_val`), `RMM2` applies **`+1`** to the new value first before applying **`-1`** to the old one. |
| 35 | 3. **Upon exiting the function, local variables that are NOT returned receive `-1`**: if a structure's counter reaches `0`, its memory is freed on the spot and **it cascades `-1`, one by one, to each of its direct children**. |
| 36 | 4. **The value returned by the function is exempt in transit**: it receives its local-variable `-1` without being destroyed as it crosses the return boundary, so when the caller writes `c = my func ()`, the `+1` from that assignment leaves its counter at `1`. |
| 37 | |
| 38 | ```ridgun |
| 39 | choose Bool [Int] [Int] => [Int] |
| 40 | choose cond a b = |
| 41 | | cond = a |
| 42 | | otherwise = b |
| 43 | |
| 44 | rmm2 example Bool => Int |
| 45 | rmm2 example cond = |
| 46 | a = [10, 20, 30] -- M1 (rc = 1) |
| 47 | b = [40, 50, 60] -- M2 (rc = 1) |
| 48 | c = choose cond a b -- `c` gets +1; the unchosen list dies right here (rc -> 0) |
| 49 | (count a) + (count c) -- On exit, `a` and `c` get -1 and 0 live blocks remain |
| 50 | ``` |
| 51 | |
| 52 | --- |
| 53 | |
| 54 | ## 8.2. Quota-Bounded Lexical Regions (`with region`): $O(1)$ Memory Firewalls |
| 55 | |
| 56 | Alongside `RMM1` and `RMM2`, whenever you are parsing an external network packet, a massive file, or a batch of untrusted data and want to enforce a **strict RAM ceiling**, you wrap the computation in a **`with region`** block: |
| 57 | |
| 58 | ```ridgun |
| 59 | process frame String => Result |
| 60 | process frame raw = |
| 61 | with region 64_KB do |
| 62 | clean = map clean line (split lines raw) |
| 63 | count valid clean |
| 64 | ``` |
| 65 | |
| 66 | 1. **Hardware/runtime-guaranteed quota**: if code inside the block attempts to allocate even a single byte beyond `64_KB` (`65,536` bytes), the runtime cleanly aborts the block, frees the entire temporary arena in constant $O(1)$ time, and returns `Err OutOfMemory` without crashing the process. |
| 67 | 2. **Instantaneous deallocation on exit**: the moment the `with region` block finishes, all temporary memory in the arena vanishes in one stroke with zero fragmentation. |
| 68 | |
| 69 | --- |
| 70 | |
| 71 | ## 8.3. Native Tensors, Einstein Contraction (`@`), and Symbolic Dimensions |
| 72 | |
| 73 | If you have ever programmed in Python with NumPy or PyTorch, you have almost certainly suffered the classic crash 41 minutes into a training run: |
| 74 | |
| 75 | ```text |
| 76 | RuntimeError: mat1 and mat2 shapes cannot be multiplied (128x64 and 32x128) |
| 77 | ``` |
| 78 | |
| 79 | In Ridgun, **`Tensor`** values are first-class native citizens and their dimensions are part of the type signature: |
| 80 | |
| 81 | ```ridgun |
| 82 | -- If (N × K) and (K × M) go in, (N × M) MUST come out! |
| 83 | multiply matrices Tensor[N, K] Tensor[K, M] => Tensor[N, M] |
| 84 | multiply matrices a b = a_ik @ b_kj |
| 85 | ``` |
| 86 | |
| 87 | Notice three standout details: |
| 88 | 1. **Static dimension verification (`Tensor[N, K]`)**: the inner axis `K` of the first matrix must match the leading axis `K` of the second. If you try to pass incompatible dimensions, `ridgunc check` stops you before executing a single instruction. |
| 89 | 2. **Einstein index notation (`a_ik @ b_kj`) and Hadamard broadcasting (`a_ij * b_j`)**: |
| 90 | - With **`@`**, indices repeated across all operands (`k`) are contracted (summed over) while unique indices (`i`, `j`) survive in the output tensor. |
| 91 | - With **`+`, `-`, `*`, `/`**, indices align dimensions and perform automatic *broadcasting* (`a_ij * b_j` multiplies each row `i` of `a` by vector `b_j` without needing `unsqueeze` or `reshape`). |
| 92 | 3. **Typed Tensors (`Tensor[T * d1 * d2]`), Mathematical Subscripts `t_(i, j)`, Slice Views `t_(i,)`, and *In-Place* Axis Mapping (`in`)**: |
| 93 | You can allocate zero-initialized tensors with explicit element types (`U64`, `Int`, `Float`, `Byte`), index them with mathematical subscripts `t_(i, j)`, take zero-copy row (`t_(i,)`) or column (`t_(,j)`) views, destructure multi-return calls directly into tensor elements with zero heap allocations, and map functions in-place along an axis using `map ... in` or `||map ... in`: |
| 94 | ```ridgun |
| 95 | fn :: blamka p16 Tensor[U64 * 16] => () |
| 96 | blamka p16 v = |
| 97 | v_(0), v_(4), v_(8), v_(12) = blamka g v_(0) v_(4) v_(8) v_(12) |
| 98 | |
| 99 | fn :: compress block Tensor[U64 * 8 * 16] Tensor[U64 * 8 * 16] => Tensor[U64 * 8 * 16] |
| 100 | compress block x y = |
| 101 | r_ij = x_ij xor y_ij |
| 102 | z_ij = r_ij |
| 103 | ||map blamka p16 in z_(i,) |
| 104 | ||map blamka p16 in z_(,j) |
| 105 | z_ij xor r_ij |
| 106 | ``` |
| 107 | |
| 108 | --- |
| 109 | |
| 110 | ## 8.4. Hardware Access (`hardware ::`) and POSIX Systems Primitives |
| 111 | |
| 112 | The `ridgunc` compiler itself, the `stable` build manager, and Unix systems utilities are written in Ridgun. To make that possible, the language includes memory-mapped register blocks (`hardware ::`) and direct POSIX primitives: |
| 113 | |
| 114 | ```ridgun |
| 115 | hardware :: UART0 @ 0x1000_0000 |
| 116 | tx_data : Int @ 0x00 |
| 117 | status : Int @ 0x04 |
| 118 | ``` |
| 119 | |
| 120 | And for interacting with the operating system: |
| 121 | |
| 122 | | Primitive | Signature | What It Does | |
| 123 | |---|---|---| |
| 124 | | `cli args` | `=> [String]` | Command-line arguments (`argv`). | |
| 125 | | `read file` / `write file` | `String => String` / `String String => ()` | Full file reading and writing on disk. | |
| 126 | | `read bytes` / `write bytes` | `String => [Int]` / `String [Int] => ()` | Exact byte-by-byte binary reading and writing (`0..255`). | |
| 127 | | `list dir` / `file lstat` | `String => [String]` / `String => ...` | Directory listing and POSIX metadata inspection (`lstat`). | |
| 128 | | `sha256 file` / `sha256 str` | `String => String` | Native 64-character hex SHA-256 cryptographic hash. | |
| 129 | | `sha256 raw` / `sha256 bytes` | `String => String` | Raw 32-byte binary SHA-256 digest (supports embedded `0x00` bytes with zero hex conversion overhead). | |
| 130 | | `sys exec` / `shell exec` | `... => Int` | Operating system process execution. | |
| 131 | | `sys getcwd` / `sys chdir` | `=> String` / `String => Int` | Query or change the current working directory. | |
| 132 | | `sys getenv` / `sys setenv` | `String => String` / `String String => Int` | Process environment variables. | |
| 133 | | `sys getuid` / `sys getgid` | `=> Int` | Real process UID and GID (`doas`, security tools). | |
| 134 | | `sys exit` | `Int => ()` | Terminate the process with an exit code. | |
| 135 | |
| 136 | --- |
| 137 | |
| 138 | ## 8.5. Real-World Case Study: Building `rg-wc`, a Parallel Unix Utility |
| 139 | |
| 140 | Let's bring together everything we have learned from Chapter 01 to Chapter 08 in a real systems program: a parallel version of the Unix `wc` utility (line and byte counter across multiple files) written in pure Ridgun: |
| 141 | |
| 142 | ```ridgun |
| 143 | -- 1. Model file metrics using an ADT record (Chapter 06) |
| 144 | type Metrics = file: String, lines: Int, bytes: Int |
| 145 | |
| 146 | -- 2. Count newline characters (ASCII code 10) with TCO tail recursion (Chapter 04) |
| 147 | fn :: count newlines String Int Int Int => Int |
| 148 | count newlines text pos end acc = |
| 149 | | pos >= end = acc |
| 150 | | (str char at text pos) == 10 = count newlines text (pos + 1) end (acc + 1) |
| 151 | | otherwise = count newlines text (pos + 1) end acc |
| 152 | |
| 153 | fn :: line count String => Int |
| 154 | line count text = count newlines text 0 (str len text) 0 |
| 155 | |
| 156 | -- 3. Analyze a file and return its Metrics record |
| 157 | fn :: analyze file String => Metrics |
| 158 | analyze file path = |
| 159 | content = read file path |
| 160 | || line count content -> n_lines |
| 161 | || str len content -> n_bytes ||. |
| 162 | Metrics path n_lines n_bytes |
| 163 | |
| 164 | -- 4. Format each row and sum totals using `fold` and `||` (Chapter 07) |
| 165 | fn :: format metric Metrics => String |
| 166 | format metric m = "{lines m}\t{bytes m}\t{file m}" |
| 167 | |
| 168 | fn :: sum lines Int Metrics => Int |
| 169 | sum lines acc m = acc + (lines m) |
| 170 | |
| 171 | fn :: sum bytes Int Metrics => Int |
| 172 | sum bytes acc m = acc + (bytes m) |
| 173 | |
| 174 | fn :: generate report [String] => String |
| 175 | generate report paths = |
| 176 | metrics = ||map analyze file paths |
| 177 | || map format metric metrics -> rows |
| 178 | || fold sum lines 0 metrics -> total_l |
| 179 | || fold sum bytes 0 metrics -> total_b ||. |
| 180 | "{rows}\nTotal: {total_l} lines, {total_b} bytes" |
| 181 | |
| 182 | fn :: main => () |
| 183 | main = |
| 184 | echo (generate report ["README.md", "AGENTS.md"]) |
| 185 | ``` |
| 186 | |
| 187 | Pause for a second and look at those ~40 lines of code: |
| 188 | - There is not a single `null`, unchecked exception, or `Mutex`. |
| 189 | - `count newlines` compiles down to a native `loop` using 0 bytes of stack thanks to automatic TCO. |
| 190 | - `||map analyze file paths` processes files in parallel across all cores of your CPU, and the `||` block computes the formatted rows, total lines, and total bytes simultaneously. |
| 191 | - Temporary memory is freed deterministically via **`RMM1`/`RMM2`**. |
| 192 | - And when you run `ridgunc build rg_wc.rg -o rg-wc`, you get a self-contained ~350 KB native binary ready to copy into `/usr/local/bin`. |
| 193 | |
| 194 | --- |
| 195 | |
| 196 | ## 8.6. Final Roadmap: The 10 Golden Rules of the Ridgun Programmer |
| 197 | |
| 198 | 1. **Always write the contract (`=>`) first and the equations (`=`) second:** thinking about types first makes the compiler work for you. |
| 199 | 2. **Let `stable format` typeset your tables:** type with single spaces and let the tool align columns, `=>`, `=`, and `->`. |
| 200 | 3. **Remember the universal `=` rule:** after any `=` you can open a sequential block, a `|` guard tree, or a parallel `||` graph. |
| 201 | 4. **Use `|.` and `||.` only when closing a nested block to return to an outer level:** keep deep decision trees compact and unambiguous. |
| 202 | 5. **Immutable by default (`x = ...`), explicitly mutable (`mut x = ...`):** reach for `mut` only when a local accumulator genuinely calls for it. |
| 203 | 6. **Use clear lowercase boolean operators (`and`, `or`, `xor`, `not`) and always close your guards with `| otherwise =`.** |
| 204 | 7. **Model your domain with ADTs (`type`):** use records with `with field: value` and sum types (`Option`, `Result`) instead of `null`. |
| 205 | 8. **Parallelize by declaring dependencies with `|| expr -> var` and `||map` / `||filter`:** let the compiler verify acyclicity and distribute work across cores. |
| 206 | 9. **Give your Tensors symbolic dimensions (`Tensor[N, K]`) and your contractions explicit indices (`a_ik @ b_kj`).** |
| 207 | 10. **Trust `RMM` (`RMM1`/`RMM2`), bound untrusted inputs with `with region`, and always verify with `stable check` and `stable test`.** |
| 208 | |
| 209 | Happy hacking in Ridgun! |