- Number
-
0010
- Title
-
The Rust Port: Workspace, Crates, Dependency Policy and Performance Parity
- State
-
committed
- Status
-
Committed
- Type
-
Architectural Specification
- Authors
-
Vikrant Rathore
- Created
-
2026-10-15
- Updated
-
2026-10-16
- Discussion
-
No external discussion URL assigned
- Summary
-
Ports Sibuna from Zig 0.16 to a Rust workspace that keeps every wire format bit for bit, builds the data plane from small in-house crates without external dependencies, and is accepted when it matches or exceeds the Zig daemon in the benchmarks of SID 0002.
- Labels
-
architecture, rust, performance
The Rust Port: Workspace, Crates, Dependency Policy and Performance Parity¶
Abstract¶
Sibuna was written in Zig 0.16. We port it to Rust. The Zig source is the specification of behaviour, and SID 0001 to SID 0009 remain the design records; this record says how their mechanisms map onto a Rust workspace. The port keeps every wire format bit for bit, so a Rust node and a Zig node that share a seed accept each other’s challenges and session tokens. The data plane depends on no external crate: hashing, signatures, base64, JSON, DEFLATE, Aho-Corasick and the HTTP/1.1 parser are small workspace crates that other projects may use on their own. Storage is the one exception, and SID 0011 records it. The port is accepted when it passes the ported test suites and matches or exceeds the Zig daemon in the throughput and latency benchmarks of SID 0002.
Motivation and scope¶
Zig 0.16 served the first version well. Its standard library carried SHA-256 with SHA-NI, BLAKE3, Ed25519, base64, JSON and a TLS client, so the daemon needed almost nothing else. Rust brings an ownership model that turns the Zig comments “this slice borrows the connection buffer until the next read” into lifetimes the compiler checks, a mature ecosystem for the one large problem we do not want to own (SQL storage), and a broader pool of maintainers.
The scope is the whole repository: the data plane (SID 0002 to SID 0004, SID 0006, SID 0009), the browser solver, storage (SID 0005, replaced by SID 0011), the console and its WebAssembly interface (SID 0007, SID 0008), the tools, the benchmarks and the documentation. Documentation moves from Typst to reStructuredText managed with guidedog: the design records with guidedog dsc and the book with guidedog build.
Specification¶
Behavioural parity¶
The following are identical to the Zig daemon, and each has tests whose expected values were produced with the Zig 0.16 standard library or taken from the Zig test suite:
- the challenge identifier (36-byte payload, 16-byte keyed BLAKE3 tag, 70 URL-safe base64 characters), the requirement ticket (59 characters), and both session token formats (75-character keyed BLAKE3 MAC token, 139-character Ed25519 token);
- the key schedule (keyed BLAKE3 subkeys
sibuna/token/v1and the others of SID 0002) and node binding of challenge keys; - Hashcash (
SHA-256(challenge ":" decimal(nonce))with bit-level difficulty) and the Cohen-Pietrzak proof layout of SID 0006; - the rule hash (Zig’s
std.hash.Wyhashwith its fixed seed), so rule identities stored in tokens agree across implementations; - the HTTP/1.1 parser’s acceptance and rejection set, including every smuggling and chunked-framing case of SID 0009;
- policy evaluation order, WEIGH scoring, WAF categories and the Elm-style diagnostics of SID 0001 and SID 0003;
- every route, status code, header and response body the end-to-end suite observes.
Workspace¶
| Crate | Reusable | Responsibility |
|---|---|---|
sibuna-crypto |
yes | SHA-1, SHA-256 with SHA-NI and ARMv8 dispatch, BLAKE3 (keyed, derive-key), HMAC, Ed25519, Wyhash, constant-time comparison; no_std |
sibuna-encoding |
yes | base64 (standard and URL-safe, with and without padding) and hexadecimal; no_std |
sibuna-json |
yes | bounded JSON parser with located errors and an escaping writer |
sibuna-pow |
yes | Hashcash and the Cohen-Pietrzak prover and verifier with a fixed workspace; no_std |
sibuna-http |
yes | zero-copy HTTP/1.1 request parser, strict chunked decoder, response writer, RFC 6455 frames and handshake |
sibuna-patterns |
yes | tagged Aho-Corasick automata over bytes |
sibuna-cidr |
yes | IPv4 and IPv6 parsing and a radix trie of prefixes |
sibuna-store |
yes | Vyukov MPSC ring, Robin Hood spent set, GCRA limiter, ban table, striped counters |
sibuna-inflate |
yes | DEFLATE and gzip with explicit output bounds |
sibuna-geoip |
yes | IP to country lookup from open CSV datasets and canonical snapshots |
sibuna-html |
yes | escaped HTML snippet rendering; templates are compiled to Rust at build time |
sibuna-core |
no | configuration, command-line parsing, diagnostics, metrics, incident types |
sibuna-policy |
no | rules, loader, evaluation engine, semantic WAF, normaliser, page templates, embeddings |
sibuna-challenge |
no | keys, tokens, challenge coordinator, tickets, adaptive difficulty |
sibuna-net |
no | streaming reverse proxy, upstream pool, forwarded metadata, WebSocket relay |
sibuna-storage |
no | persistent layer over sqlx (SID 0011) |
sibuna-serve |
yes | bounded HTTP/1.1 listener with joinable workers, deadlines, admission classes and WebSocket upgrades |
sibuna-tls |
no | blocking outbound TLS through rustls for the console (SID 0012) |
sibuna-console* |
no | console protocol, server and WebAssembly interface (SID 0007) |
Binaries live under cmd/: the daemon sibuna and the browser solver sibuna-pow-wasm. Repository tools are Rust programs under tools/, run through cargo xtask. Dependencies point one way: reusable crates, then sibuna-core, then the application crates, then the binaries.
Mapping of mechanisms¶
| Zig | Rust |
|---|---|
| slices into the 64 KiB connection buffer | &'a [u8] borrowed from the buffer; a parsed Request<'a> cannot outlive the bytes it describes, and the compiler rejects a use after the buffer is refilled |
| error unions and error sets | Result<T, E> with one enum per module implementing Display and std::error::Error |
maxInt and empty-string sentinels |
Option<T> |
| engine slot with an atomic reader count (read-copy-update) | Arc<Engine> published through a slot; the storage thread builds a new engine off the hot path and swaps it in; the old one is freed when its last request finishes |
| lock-striped fixed tables with spin locks | the same striping with std::sync::Mutex per shard (a futex in the uncontended case), unless a benchmark shows a spin lock wins |
threadlocal sampling state and counter stripe |
thread_local! with Cell |
comptime tables and the HTML snippet compiler |
const tables and const fn; snippets are compiled by a build script into Rust functions, so template errors (HTML001 to HTML007) stop the build as before |
@embedFile |
include_bytes! and include_str! |
defer and errdefer |
Drop and guard values |
| thread-per-connection with a joinable task pool | the same model on std::thread: accept workers hand each connection to a bounded pool of threads, and the slot is reclaimed by joining before reuse |
Concurrency model¶
The Zig daemon serves each connection on its own thread, bounded by --max-connections, with one 64 KiB buffer per connection. We keep this model. It is simple, it isolates a slow client to one thread, and it lets the request path stay synchronous and allocation-free. An asynchronous runtime would add a large dependency to the hot path and change the cost model of every mechanism the earlier records measured. The storage layer (SID 0011) is the only place that runs an asynchronous runtime, on its own thread, and the request path never waits on it.
Hardware acceleration¶
Zig’s standard library selected SHA-NI and the ARMv8 SHA-2 instructions at build time. We detect them at run time with is_x86_feature_detected! and is_aarch64_feature_detected! and fall back to a portable implementation, so one binary is fast on every machine it runs on. Hashcash verification, PoSW verification and the challenge proof are SHA-256 bound, so this is where hardware acceleration pays. Other SIMD use (for example a vectorised byte-class pre-scan before WAF inspection) is added only with a benchmark that shows the gain, and lives in a module of its own with // SAFETY: comments on every unsafe block.
The browser solver¶
sibuna-pow-wasm is a no_std cdylib for wasm32-unknown-unknown. It exports the same functions as the Zig module (sibuna_get_buffer_ptr, sibuna_get_buffer_len, sibuna_solve_step, sibuna_solve_sha256, sibuna_posw_solve, sibuna_posw_proof_ptr), so the worker script and the JavaScript fallback are unchanged. It compiles the same sibuna-pow sources that the server uses to verify, so prover and verifier cannot drift apart. It uses no wasm-bindgen; the module is built for size.
Dependency policy¶
Every external crate is a liability: it must be reviewed, updated and trusted. The data plane uses none. A dependency is admitted only when it is mature, self-contained and solves a genuinely large problem, and the reason is recorded next to it in Cargo.toml and in a design record. Storage is such a case: we use sqlx (SID 0011), confined to sibuna-storage behind cargo features, so cargo build --no-default-features produces a daemon with no external crate at all.
Engineering standards¶
The standards of SID 0001 carry over, written for Rust:
- A function body has at most 70 logical lines; a file at most 1,408 lines; a line at most 99 characters (
rustfmt.tomlsetsmax_width = 99).cargo xtask checkenforces the limits, andclippy.tomlsetstoo-many-lines-threshold = 70. - Every public item has a
///comment saying what it borrows, what it returns and how it fails. - No
unsafeoutside FFI and measured SIMD modules; everyunsafeblock has a// SAFETY:comment. - No
unwrap()orexpect()on input-dependent values in library code. Libraries return structured errors and never print. The binaries render Elm-style diagnostics. - Plain-data APIs: public-field structs with
Default, a small verb vocabulary (load_*,begin_*/end_*,is_*,*_count), noget_prefix. - Bounded work on untrusted input: explicit limits that return errors, never panics or unbounded allocation.
cargo build,cargo test,cargo clippy --all-targets -- -D warnings,cargo fmt --checkandcargo xtask checkpass before a change is merged. GitHub workflows may use shell or Python glue; repository tools are Rust.
Memory management that Rust makes unnecessary¶
Much Zig code existed only to manage memory by hand or to avoid footguns: deinit and errdefer chains, arenas reset per rebuild, double-buffered engine slots reused to avoid reallocating several megabytes, undefined initialisation, alignment workarounds for heap fixtures, and comments of the form “this slice must not outlive the next read”. The port does not carry this machinery over. Ownership, borrowing, Drop and values built once off the hot path give the same guarantees, checked by the compiler. Each crate’s documentation names the Zig mechanisms it removed and what replaced them. The observable behaviour (bytes on the wire, decisions, bounds and error texts) does not change, and the request path still does not allocate.
Tools and documentation¶
Repository tools are Rust programs under tools/, in the pattern guidedog uses: tools/xtask dispatches cargo xtask <task> (check, and later wasm and bench), and standalone tool crates sit beside it. The Zig repository’s Python and shell drivers are not ported as scripts. End-to-end tests become Rust integration tests, and browser checks drive headless Chromium from Rust over the DevTools protocol, reusing the workspace’s WebSocket and JSON crates. Python is allowed only in GitHub workflow glue, or in a benchmark driver when no reasonable Rust alternative exists.
Documentation is not code. The design records are a guidedog discussion catalog (docs/sid, managed with guidedog dsc) and the book and manual are guidedog projects; their PDFs use the Sibuna Typst design in each project’s _templates/. The repository contains no documentation build code: the Zig tools/sid.zig and the zig build sid* and book steps have no Rust counterpart.
The OWASP Core Rule Set¶
Sibuna’s WAF is its own structural inspection (SID 0004), not the OWASP Core Rule Set, and the port keeps it that way. We evaluated the external SecLang engines available to Rust (pure-Rust engines a few months old with a single maintainer each, and bindings to libmodsecurity or Coraza) and admit none of them: an inspection engine on an edge server is security-critical code we must own, review and maintain, as the Zig daemon owned its own. If Sibuna adds compatibility with CRS rules, it does so in-house: a new design record specifies the subset of SecLang, the transformations, the operators and their algorithms (for example the existing Aho-Corasick automata for phrase matching and linear-time matching for expressions), and how CRS anomaly scores map onto WEIGH decisions; the implementation follows that record.
Rationale and alternatives¶
We considered porting zaxonlite as well. It is a 48,000-line Zig project of its own with a consensus library; porting it would double the work and keep an unusual storage engine in a firewall whose state is small. SID 0011 explains the replacement.
We considered tokio with hyper for the data plane. The Zig daemon’s measured performance comes from a zero-copy parser, a fixed per-connection buffer and allocation-free decisions, none of which an asynchronous runtime improves; hyper would also take the request bytes away from the inspection code. We keep the thread model and measure it against the Zig daemon before considering an event loop. If the comparison shows a gap that the thread model cannot close, an epoll loop over the same parser is the next step, recorded in a new SID.
We considered wasm-bindgen for the browser modules. The solver and the console already use a small fixed bridge (shared buffers and integer exports); wasm-bindgen would add generated glue and a build tool without adding a capability.
Security and operational considerations¶
Bit-identical formats are a security property as well as a convenience: a mixed Zig and Rust cluster during migration must not accept a token one node minted under different rules. The test vectors pin each format. Constant-time comparison is used wherever the Zig code used timing_safe.eql. Secrets are zeroed on drop where the Zig code used secureZero.
Backwards compatibility¶
Command-line options, routes, the policy file format and wire formats are unchanged, except for the storage options that SID 0011 replaces. A data directory written by the Zig daemon (Zaxonlite files) is not read by the Rust daemon; operators export policies with sibuna console policies export before migrating and import them afterwards.
Acceptance criteria and implementation status¶
The port is accepted when:
- Every Zig unit test has a Rust counterpart that passes, and the end-to-end suite passes against the Rust daemon.
- Cross-implementation tests show that a Zig node and a Rust node sharing a seed verify each other’s session tokens and requirement tickets.
- The browser solver passes in a headless browser with WebAssembly and with the JavaScript fallback.
- The benchmarks of SID 0002 (primitives, and whole-product runs under
wrk) show the Rust daemon at or above the Zig daemon’s throughput, with p99 latency no worse, on the same machine; the results are recorded underbenchmarks/results/. - The console passes its golden fixtures and its browser workflows in a headless browser, with screenshots reviewed.
Status: all five criteria are met (evidence below, 2026-10-15). Committed on 2026-10-16 after the maintainer’s review; released in 1.0.0-beta.1.
Evidence recorded so far (2026-10-15):
- Criterion 2: Rust and Zig daemons sharing a seed verify each other’s MAC and Ed25519 session tokens and requirement tickets (
crates/challengecross-implementation tests against the Zig baseline binary). - Criterion 3:
cargo xtask browserdrives Chromium 155 headless through the interstitial; the challenge passes with WebAssembly (solve 29 to 34 ms) and with the JavaScript fallback (84 to 91 ms, WebAssembly disabled). Detection relies on the default interstitial markup (#status,.progress-bar). - Differential behaviour:
cargo xtask parityreplays generated corpora against both daemons in 17 configurations. At commitdda1162four seeds (three of 20,000 requests and one of 60,000) gave no unexplained difference; every allowed difference is listed with its reason intools/parity/allow-list.json. - Criterion 4, met: on an AMD Ryzen 9 5950X with the daemons on CPUs 0 to 3 and the load on CPUs 8 to 15 (
benchmarks/results/, 2026-10-15), the Rust daemon is at or above the Zig daemon’s throughput in all 16 whole-product workloads (ratios 1.00 to 1.09) with p99 no worse (ratios 0.46 to 1.00) and 4 to 18% less CPU per request; peak memory is about half (16 MiB against 31 MiB). At equal offered load from 10,000 to 150,000 requests per second, every percentile from p50 to p99.9 is at or below Zig’s, after the connection threads were made reusable (SID 0002 revision note; before that, p99.9 at 100,000 requests per second was 0.3 to 2 ms against Zig’s 0.1 ms). Of the primitive rows, all are faster than Zig exceptgcra_checkandgcra_four_rule_scopes(12.8 against 5.7 ns), whose byte API now includes the keyed SipHash-1-3 of the client (a security fix); the daemon hashes once per request, and the hashed rows measure 6.2 ns. - Criterion 5, met: the console’s 290 interface tests, including the golden fixtures, pass.
cargo xtask browser consolepassed all 17 steps against a fresh daemon in Chromium 155 (no WebSocket and no map request before sign-in, the forced password change, live tiles across stats deltas, every page with no exception, console error or CSP violation, the wall display limited to statistics, the WebSocket closed at sign-out), and we reviewed its screenshots; the review led to three interface fixes (sparkline width, the Audit and Tokens scroll position, the heading focus ring). The eight-dashboard smoke run is inbenchmarks/results/console-8-dashboards-2026-10-15.md. - Criterion 1, met: every one of the 567 named Zig tests (
test "..."blocks underlibs/andapps/) is mapped indocs/sid/assets/0010-zig-test-map.tsv: 540 have a Rust test that checks the same behaviour, 13 had none and were ported on 2026-10-15 as 11 Rust tests, and 14 do not apply, each with the record that changed or removed the mechanism (Zaxonlite and the consensus log, Zig data migrations, the comptime JSON decoder, SQL quoting replaced by bound values, and the like). One ported test found a deviation, now fixed: a retried incident batch mapped its country again instead of keeping the one it was first mapped with (SID 0013). The workspace suite (about 2,100 tests on SQLite, and 700 more against PostgreSQL) and the daemon and console end-to-end suites pass. - Differential behaviour, latest: at commit
f5fa5be,cargo xtask parity(seed 20261015, 20,018 requests, 17 configurations) gave no unexplained difference.
Open questions¶
- Closed 2026-10-15: the spent set and the limiter use
std::sync::Mutex(see the review history for the measurements); the ban table keeps its seqlock readers with a mutex for writers. - Closed 2026-10-15: no
epollevent loop is needed for parity. Blocking threads, reused across connections, meet criterion 4.
Review history¶
- 2026-10-15: drafted with the start of the port.
- 2026-10-15 (Rust port): each crate lists its deliberate changes from Zig in its crate documentation, and the design records that those changes touch carry dated revision notes (SID 0002 to SID 0007, SID 0009, SID 0011, SID 0012). Evidence for the first open question:
sibuna-storemeasuredstd::sync::Mutexagainst a spin lock on a Ryzen 9 5950X. The spin lock saves about a nanosecond per uncontended operation but loses under contention (the spent set at 64 threads: 11.2 against 6.9 million operations per second; GCRA on six hot addresses at 64 threads: 37.4 against 6.5 million), so the spent set and the limiter use the mutex; the question is closed in favour of the mutex. The HTML snippet compiler ofsibuna-htmladds the codeHTML008(a snippet file name must yield Rust item names) toHTML001toHTML007. - 2026-10-15 (security review): limiter and ban-table placement is process-local, keyed per table, and not part of any wire format; only the rule hash keeps Zig’s
std.hash.Wyhashwith its fixed seed. The Vyukov ring remains the store’s only non-FFIunsafe:std::sync::mpsc::sync_channelwas measured to cost 20 to 45% more per push with 16 producers (tests/timing.rs,bounded_queue_producers). An adversarial review (2026-10-15) found request-path weaknesses inherited from Zig (trusted forwarded address, origin pool reuse, request-target normalisation, WAF canonicalisation bound, keyed tables, honeypot cross-site requests, adaptive input, connection limits); each fix is recorded in the SID that specifies the mechanism. - 2026-10-15 (console port):
cmd/sibunafeatures aredefault = [sqlite, console];consoleenables storage and the cratessibuna-console,sibuna-console-protocol,sibuna-console-storageandsibuna-json.--no-default-featuresstays free of external crates. The console server is the application cratesibuna-console; TOTP joinssibuna-cryptoand the range-to-prefix split joinssibuna-geoip, both reusable. - 2026-10-15 (console port):
sibuna-cryptogainstotp(RFC 4226 and 6238, HMAC-SHA1, six digits, 30 s steps, base32 seeds) for the console’s second factor;sibuna-geoip::prefixesholds the range-to-prefix split used by country actions. - 2026-10-15 (console port):
sibuna-tlshas atestingfeature,testing::Ingress, a TLS-terminating test proxy with the fixed test certificates; only dev-dependencies enable it. - 2026-10-15 (console port):
sibunahas an optional build-dependency onsibuna-geoip(console feature) to validate the embedded GeoIP snapshot. The console’s end-to-end suite iscmd/sibuna/tests/console_*.rs; its PostgreSQL cluster scenarios run whenSIBUNA_TEST_DATABASE_URLis set. With the console compiled in, disabled, storage only, or enabled and idle, the 16 product workloads stay within about 1% of the daemon built without it, which is within noise.
References¶
- SID 0001: the Sibuna Discussion process and engineering standards.
- SID 0002: foundation architecture and performance contract.
- SID 0006: mathematical foundations.
- SID 0009: chunked request bodies.
- SID 0011: storage through sqlx.
- The Rust Reference and the Rust standard library documentation,
std::arch(runtime feature detection).
Lifecycle event
2026-10-15: prediscussion -> discussion. Specification of the port, drafted
before the code
Lifecycle event
2026-10-16: discussion -> accepted. Implemented and released in Sibuna
1.0.0-beta.1; reviewed by the maintainer on 2026-10-16.
Lifecycle event
2026-10-16: accepted -> committed. Implemented and released in Sibuna
1.0.0-beta.1; reviewed by the maintainer on 2026-10-16. All five acceptance
criteria met, with the evidence recorded in the record (test map
docs/sid/assets/0010-zig-test-map.tsv, cross-implementation token tests,
cargo xtask browser, benchmarks/results); release v1.0.0-beta.1