Ridgun
The language itself: a compiler that compiles itself, the stable project tool, the specification and the book
| 1 | # Chapter 07: Higher-Order Functions and Sweat-Free Parallelism (No `Mutex`, No `Deadlocks`) |
| 2 | |
| 3 | In almost every programming language there is a dramatic moment in the life of a project—the day someone says in a meeting: |
| 4 | |
| 5 | > *"This is running slow. We need to parallelize it with threads."* |
| 6 | |
| 7 | Silence falls over the room. Everyone knows what comes next: weeks chasing phantom *race conditions* that only happen on Tuesdays during a full moon, *deadlocks*, and irreproducible bugs. |
| 8 | |
| 9 | In Ridgun, when someone says *"we need to parallelize this"*, you simply prefix your list operations with `||` (**`||map`** and **`||filter`**) or drop your calculations into a parallel `||` block with `->` arrows and go grab a coffee. |
| 10 | |
| 11 | Why? Because **when your functions are pure and your data is immutable by default, two CPU cores working side by side can never step on each other's toes**. |
| 12 | |
| 13 | Let's walk through it step by step. |
| 14 | |
| 15 | --- |
| 16 | |
| 17 | ## 7.1. Functions That Take Functions: `map`, `||map`, `filter`, `||filter`, and `fold` |
| 18 | |
| 19 | In Ridgun a function is just another value. You can pass it as an argument the exact same way you pass the number `7`. |
| 20 | |
| 21 | The language comes out of the box with collection combinators so you can transform lists without writing a single manual recursion, both in sequential mode (`map`, `filter`, `fold`) and in multicore parallel mode using the `||` prefix (**`||map`**, **`||filter`**): |
| 22 | |
| 23 | ### 1. `map` (sequential) and `||map` (multicore parallel) |
| 24 | Suppose you have a function `expensive Int => Int` and a list of numbers: |
| 25 | |
| 26 | ```ridgun |
| 27 | expensive Int => Int |
| 28 | expensive x = (x * x) + (x * 3) |
| 29 | |
| 30 | main => () |
| 31 | main = |
| 32 | data = [1, 2, 3, 4, 5, 6, 7, 8] |
| 33 | -- Strict left-to-right sequential: |
| 34 | r1 = map expensive data |
| 35 | -- Multicore parallel with the `||` prefix: |
| 36 | r2 = ||map expensive data |
| 37 | echo r1 |
| 38 | echo r2 |
| 39 | ``` |
| 40 | |
| 41 | - With **`map`**, Ridgun applies the function element by element from left to right on the current thread (ideal for small lists or functions with ordered I/O side effects). |
| 42 | - With **`||map`**, the Ridgun runtime distributes the elements of `data` across your processor's worker threads and reassembles the final list **in the exact same order**, guaranteeing that `||map f xs == map f xs`. |
| 43 | |
| 44 | ### 2. `filter` (sequential) and `||filter` (parallel): Keeping only elements that match a condition |
| 45 | You pass a function that returns `Bool` (a *predicate*) and a list: |
| 46 | |
| 47 | ```ridgun |
| 48 | is positive Int => Bool |
| 49 | is positive n = n > 0 |
| 50 | |
| 51 | main => () |
| 52 | main = |
| 53 | clean = filter is positive [-5, 10, 0, -2, 42] |
| 54 | clean_par = ||filter is positive [-5, 10, 0, -2, 42] |
| 55 | echo clean -- Prints [10, 42] |
| 56 | echo clean_par -- Prints [10, 42] |
| 57 | ``` |
| 58 | |
| 59 | ### 3. `fold`: Reducing an entire list down to a single result |
| 60 | You pass a two-argument combining function `(Accumulator Element => Accumulator)`, the initial value, and the list: |
| 61 | |
| 62 | ```ridgun |
| 63 | multiply acc Int Int => Int |
| 64 | multiply acc acc x = acc * x |
| 65 | |
| 66 | main => () |
| 67 | main = |
| 68 | echo (fold multiply acc 1 [2, 3, 4, 5]) -- 1 * 2 * 3 * 4 * 5 = 120 |
| 69 | ``` |
| 70 | |
| 71 | ### How Do I Write My Own Function That Accepts Another Function? |
| 72 | Very easily: in the type signature, wrap the guest function's signature in parentheses `(Input => Output)`: |
| 73 | |
| 74 | ```ridgun |
| 75 | run twice (A => A) A => A |
| 76 | run twice f x = f (f x) |
| 77 | ``` |
| 78 | |
| 79 | ### 4. Idiomatic Left-to-Right Pipelines (`->`) — The 100% Ridgun Way! |
| 80 | When you chain several transformations (`map`, `filter`, `fold`, or your own functions and `op ::` operators), nesting parentheses from right to left (`compact l (filter g (map f xs))`) forces you to read inside-out. |
| 81 | |
| 82 | In Ridgun, the **recommended idiomatic style (`2b`)** reads from left to right using the pipeline operator **`->`**, where **whatever comes from the left always enters as the first argument** of the next function or `op ::`: |
| 83 | |
| 84 | ```ridgun |
| 85 | -- Recommended idiomatic Ridgun style (2b): |
| 86 | xs map (x => x * 2) -> filter (x => x > 10) -> fold (acc x => acc + x) 0 -> total |
| 87 | |
| 88 | -- And if a function returns a tuple, filter it with a mask and unpack it into the next stage: |
| 89 | parse packet "v0.9.9" -> (+ _ _ +) -> combine code -> summary |
| 90 | ``` |
| 91 | |
| 92 | Plus, any expression with side effects (writing a file, invoking a system call) can stand on its own line without `echo` or dummy `_ =` variables—or end with `-> ()` when you want to discard its return value explicitly. |
| 93 | |
| 94 | ### 5. Lexical Closures and Anonymous Functions (`(x => expr)`, `(acc x => expr)`) |
| 95 | When you want to pass a short function on the fly or capture local variables from the enclosing scope (such as an offset, a prefix, or an active session) without declaring a separate top-level helper, use a parenthesized **lexical closure** with **`=>`**: |
| 96 | |
| 97 | ```ridgun |
| 98 | fn :: process with session => [Int] |
| 99 | process with session = |
| 100 | offset = 10 |
| 101 | bonus = 3 |
| 102 | added = map (x => x + offset) [1, 2, 3] |
| 103 | filtered = filter (x => x >= bonus) [1, 2, 3, 4] |
| 104 | total = fold (acc x => acc + x + bonus) 0 [1, 2, 3] |
| 105 | pairs = map ((a b) => a + b + offset) [(1, 2), (3, 4)] |
| 106 | added :: filtered :: [total] :: pairs |
| 107 | ``` |
| 108 | |
| 109 | Five rules keep lambdas clean and unambiguous: |
| 110 | - **Mandatory parentheses**: every lambda is enclosed in parentheses `(... => ...)`. |
| 111 | - **Zero-parameter lambdas (`(=> expr)`)**: just as `fn :: main => ()` forbids `()` before `=>`, a zero-parameter lambda is written `(=> expr)` (and its type is `(=> Type)`), never `(() => expr)`. |
| 112 | - **Multiple parameters vs. tuple (`(a b => ...)` vs. `((a b) => ...)`)**: `(acc x => ...)` takes two separate parameters, whereas `((a b) => ...)` takes a single tuple and destructures it. |
| 113 | - **Multi-line bodies**: placing a newline after `=>` lets you write an indented multi-step block before the closing `)`: |
| 114 | ```ridgun |
| 115 | doubled = map (x => |
| 116 | step = x * 2 |
| 117 | step + offset |
| 118 | ) [1, 2, 3] |
| 119 | ``` |
| 120 | - **Uppercase vs. lowercase**: `(Conn => Conn)` is a function type (uppercase initial) and `(conn => conn)` is a lambda value (lowercase initial). |
| 121 | |
| 122 | Captured bindings (`offset`, `bonus`) are automatically lifted with deterministic `RMM1`/`RMM2` reference management—zero memory leaks and zero garbage-collector pauses. |
| 123 | |
| 124 | --- |
| 125 | |
| 126 | ## 7.2. The Crown Jewel: `||` Blocks with `->` as Parallel Graphs (DAGs) |
| 127 | |
| 128 | This is where Ridgun changes how you think. |
| 129 | |
| 130 | In an ordinary imperative language, when you write three lines in a row: |
| 131 | 1. First line 1 runs. |
| 132 | 2. Then line 2 runs (even if it has nothing to do with line 1). |
| 133 | 3. Then line 3 runs. |
| 134 | |
| 135 | That forces your computer to work wearing blinders. In Ridgun, when you use parallel bars **`||`** with arrows **`->`** (without needing any extra keyword like `where`), you are not issuing sequential commands: you are drawing a **Directed Acyclic Graph (DAG)** that declares which computation feeds which variable: |
| 136 | |
| 137 | ```ridgun |
| 138 | fib Int => Int |
| 139 | fib 0 = 0 |
| 140 | fib 1 = 1 |
| 141 | fib n = (fib (n - 1)) + (fib (n - 2)) |
| 142 | |
| 143 | fact Int => Int |
| 144 | fact 0 = 1 |
| 145 | fact n = n * (fact (n - 1)) |
| 146 | |
| 147 | super calc => Int |
| 148 | super calc = left_branch + right_branch |
| 149 | || fib 10 -> left_branch |
| 150 | || fact 6 -> right_branch |
| 151 | ``` |
| 152 | |
| 153 | Read it out loud: |
| 154 | - *"I want to compute `left_branch + right_branch`, while in parallel (`||`) `fib 10` flows into `left_branch` (`->`) and `fact 6` flows into `right_branch` (`->`)."* |
| 155 | |
| 156 | ### What `ridgunc` Does Behind the Scenes with Your `||` Lines: |
| 157 | 1. It inspects the `->` arrows and builds the dependency graph (DAG). |
| 158 | 2. **It catches cycles statically**: if by mistake you write `|| a + 1 -> b` and `|| b + 1 -> a`, the compiler stops you on the spot with *"Hold on! There is a circular dependency"* and refuses to compile an infinite hang. |
| 159 | 3. It sees that `fib 10` and `fact 6` do not depend on each other, so they get the green light to run in parallel before being added in `left_branch + right_branch`. |
| 160 | |
| 161 | You can even build a multi-stage pipeline inside the same `||` block—either attached to the bottom of an expression or right after `=`—and the compiler will figure out all by itself which stages can run simultaneously and which ones must wait: |
| 162 | |
| 163 | ```ridgun |
| 164 | quarterly balance Int => Int |
| 165 | quarterly balance base = |
| 166 | || base * 100 -> gross_sales |
| 167 | || base * 40 -> fixed_costs |
| 168 | || gross_sales - fixed_costs -> gross_profit ||. |
| 169 | gross_profit * 80 / 100 |
| 170 | ``` |
| 171 | |
| 172 | Notice the **`||.`** terminator at the end of the last `||` line: just as `|.` closes a nested guard tree, **`||.`** closes the parallel `||` block when normal sequential statements continue below it. Here `gross_sales` and `fixed_costs` start in parallel; once both finish, `gross_profit` is computed, and finally `gross_profit * 80 / 100` is returned. All without a single `async`, `Future`, `Promise`, or `Mutex`. |
| 173 | |
| 174 | --- |
| 175 | |
| 176 | ## 7.3. The Nameless `||` Shortcut: Parallel Pipelines That Return a Tuple |
| 177 | |
| 178 | What if you just want to launch two or three calculations in parallel and collect them together into a tuple without even inventing variable names with `->`? |
| 179 | |
| 180 | Use `||` directly: |
| 181 | |
| 182 | ```ridgun |
| 183 | parallel pair Int => (Int, Int) |
| 184 | parallel pair n = |
| 185 | || fib n |
| 186 | || fact n |
| 187 | ``` |
| 188 | |
| 189 | Direct, visual, and it returns the tuple `(fib n, fact n)`. |
| 190 | |
| 191 | --- |
| 192 | |
| 193 | ## 7.4. When You Want Manual Steering: the Native Agent Model (`agent`, `send ... to`, `receive from`) and CSP Channels |
| 194 | |
| 195 | Automatic parallelism via `||` blocks and `||map` covers 90% of real-world cases. But when you are building a persistent server, isolated user sessions, a job queue, or a producer-consumer pipeline, you want **concurrent agents with isolated memory and statically typed message passing**. |
| 196 | |
| 197 | In Ridgun there is no need for a special `actor ::` declaration keyword: **any normal function (`fn ::` or `op ::`) becomes a concurrent agent in a single line with `agent`**: |
| 198 | |
| 199 | ### 1. Stateless and stateful agents (`agent`, `send ... to`, `receive from`, `stop`) |
| 200 | - `ag = agent (my function)`: wraps a function `M => R` into a stateless agent of type `Agent M R`. |
| 201 | - `ag = agent (stateful function) initial_state`: wraps a function `S M => S` (or `S M => S R`) into an agent with isolated state that remembers its state across messages without a single `Mutex`. |
| 202 | - `send msg to ag` (or with multiple arguments: `send a b to ag`): sends a typed message to the agent's inbox. |
| 203 | - `res = receive from ag` (or non-blocking `try receive from ag`): pulls the next reply from the agent's outbox. |
| 204 | - `stop ag`: terminates the agent and deterministically frees its state via `RMM1`/`RMM2`. |
| 205 | |
| 206 | ```ridgun |
| 207 | fn :: add visit Int Int => Int |
| 208 | add visit acc n = acc + n |
| 209 | |
| 210 | fn :: process batch Int Int => Int |
| 211 | process batch id t = (id * 100) + t |
| 212 | |
| 213 | fn :: main => () |
| 214 | main = |
| 215 | counter = agent (add visit) 10 |
| 216 | send 5 to counter |
| 217 | send 7 to counter |
| 218 | v1 = receive from counter |
| 219 | v2 = receive from counter |
| 220 | stop counter |
| 221 | |
| 222 | worker = agent (process batch) |
| 223 | send 3 25 to worker |
| 224 | batch = receive from worker |
| 225 | stop worker |
| 226 | echo "Visits:" v1 "->" v2 "| Batch:" batch |
| 227 | ``` |
| 228 | |
| 229 | ### 2. Smart compile-time strategy selection (*Green Threads* vs. Real OS Kernel Threads) |
| 230 | A common question is: *"Do `||`, `||map`, and `agent` always spawn OS threads, or do they run as lightweight green tasks?"* |
| 231 | |
| 232 | In Ridgun, **the compiler makes that decision for you by statically analyzing each function**: |
| 233 | - **Lightweight functions (`TierTrivial` / `StratInlineSequential`)**: If the function passed to `||`, `||map`, `||filter`, or `agent` has shallow depth (arithmetic, string formatting, fast HTTP session state transitions), warming up OS threads would cost more than the computation itself. The compiler executes it as an ultra-lightweight *green-thread* step with zero `SYS_clone` overhead. |
| 234 | - **Heavy or deeply recursive computations (`TierDynamicRecursive` / `TierHeavyCPU`)**: If the function has deep recursion (like `fib` or `fact`) or tensor algebra (`einsum`, `hadamard`, `@`), the compiler automatically emits multicore dispatch across **real OS kernel threads (`SYS_clone`)**, processing `||` branches or queued `agent` messages in true hardware parallelism across your CPU cores. |
| 235 | |
| 236 | ### 3. Passing messages through bounded channels (`new bounded channel`) |
| 237 | Just like in Go or Erlang, in addition to agents you can also create explicit thread-safe pipes to connect pipeline stages: |
| 238 | |
| 239 | - `new bounded channel capacity`: creates a channel with room for `capacity` messages. |
| 240 | - `send through value channel` (or `chan send value channel`): puts a message into the pipe. |
| 241 | - `receive from channel` (or `chan recv channel`): pulls the next message from the pipe. |
| 242 | - `try receive from channel`: checks whether a message is waiting without blocking; returns `Some value` or `None`. |
| 243 | - `receive from timeout channel ticks`: waits up to a time limit and returns `Ok value` or `Err Timeout`. |
| 244 | |
| 245 | ```ridgun |
| 246 | fn :: main => () |
| 247 | main = |
| 248 | pipe = new bounded channel 4 |
| 249 | _ = send through "Task A completed" pipe |
| 250 | _ = send through "Task B completed" pipe |
| 251 | first = receive from pipe |
| 252 | second = receive from pipe |
| 253 | echo first "|" second |
| 254 | ``` |
| 255 | |
| 256 | ### 4. Fire-and-forget sink agents (`agent sink`), global mailboxes (`agent ::`), and deterministic testing (`with agent`) |
| 257 | When you need asynchronous audit logs, metrics, tracing, or cross-cutting rate limiting without threading an extra handle through every function signature or accumulating unread replies in an output queue, Ridgun provides **sink agents (`agent sink`)** and **top-level global mailboxes (`agent ::`)**: |
| 258 | - `rec = agent sink (step fn) [init_state], mailbox N drop|block, drain 2_s`: processes messages in FIFO order (`S M => S` or `M => ()`), releases old states via `RMM1`/`RMM2`, and has **no output queue** (`receive from`, `try receive from`, and `ask` on a `SinkAgent` are compile-time errors). |
| 259 | - `agent :: audit_log = sink (step fn) [], mailbox 10_000 drop, drain 2_s`: declares a module-level sink agent initialized before `main` and drained in reverse declaration order at program exit. |
| 260 | - `agent dropped ag`, `agent flush ag`, `agent state ag`, and `with agent audit_log = spy do ...`: let you inspect backpressure drops, flush pending messages, inspect state snapshots, and redirect a global mailbox to an isolated spy agent in tests. |
| 261 | - In addition, `Channel`, `Agent`, and `SinkAgent` descriptors can be shared across multiple workers (`send ch 1 to w1`, `send ch 2 to w2`), while regular heap payloads sent through channels remain affine (`ErrUseAfterSend`). |
| 262 | |
| 263 | And now, fasten your seatbelt for **Chapter 08**, the grand finale: **tensors with compile-time verified dimensions, scoped variables (`context`/`shared`), deterministic memory regions, direct MMIO hardware access, and UNIX/POSIX systems programming**. |