ZE-020 - Standard Library

Basic Standard Library

Introduction

Adds a basic set of helpers to the standard library modules.

Motivation

Currently Zirric lacks the most basic operations regarding prelude types. For example, it is impossible to actually work with Binary or String. Furthermore it is currently impossible to actually write tests in Zirric.

Proposed Solution

Add a new set of modules around every logical group of types:

• arrays provides helpers for prelude.Array
• bytes for working with prelude.Byte and prelude.Binary
• clock reads the time, kept apart from time so that nothing pure depends on a clock
• dicts supports prelude.Dict
• errors is centered around @prelude.Error
• fmt no longer depends on os
• fs abstracts a filesystem, with an in-memory one so that code touching files stays testable
• fun holds function-level helpers and lazy sequence operations
• io adds new attributes for dependency injection
• math covers prelude.Int and prelude.Float, which had no operations beyond the arithmetic operators
• options adds helpers around prelude.Option and @prelude.AnyOption
• os hands out the host’s own capabilities: its filesystem, its two clocks, its two random sources and the standard streams
• paths manipulates slash-separated paths as text, without touching a filesystem
• prelude got some slight adjustments
• random supplies randomness as a value, reproducibly; the fast and cryptographic sources are handed out by os
• ranges covers prelude.Range, prelude.ClosedRange and prelude.OpenRange
• reflect inspects modules, types and their fields, with reflect.packages reaching the modules of a package
• results adds helpers around prelude.Result and @prelude.AnyResult
• scripts collects the conveniences that assume a process, such as printing to stdout — it ships as its own package rather than with the standard library
• strings for working with prelude.Char and prelude.String
• time supplies Duration, Instant and Timestamp as primitives, with no way to read a clock
• tests makes it possible to write tests in Zirric, with tests.runner discovering and running them through reflect

Detailed Design

One module per type group, named after the domain

Every module is named after the group of prelude types it serves, in the plural, and its directory name matches its mod line one-to-one. Byte and Binary share bytes rather than being split into bytes and binaries, because a Binary is a sequence of Byte and splitting them would force an arbitrary call on every future helper. The same reasoning keeps Char and String together in strings.

Modules are embedded into the compiled binary through an embed.go per directory and a corresponding entry in DefaultStdlibProvider, so they are always available without being declared as a dependency.

The Like unions

Several modules accept more than one concrete type for the same conceptual argument. Rather than overloading, each such module declares a Like union and a from(Like) normalizer used by every other function in that module:

This keeps bytes.contains(someBinary, 'x') and bytes.contains(someBinary, otherBinary) a single declaration.

arrays deliberately has no Like union: an Any | Array union would not be meaningful, since any value can already be an array element.

Zirric or Go

Helpers are written in Zirric wherever the language can express them, and as extern fn only where it cannot, or where doing so in Zirric would be quadratic:

• bytes and arrays are mostly Zirric. Binary and Array both support append, which grows amortized in constant time, so building a result incrementally is efficient. Only the operations needing raw byte access — reinterpretation, the hex codec, slicing and substring search — are extern fn.
• strings is mostly Go. Character indexing has to decode UTF-8, and repeated result = result + part is quadratic because strings are immutable, so anything that builds a string from parts is implemented with a Go strings.Builder instead.
• dicts, errors, fun, options, ranges and results are pure Zirric.
• math is almost entirely Go, since the VM implements the arithmetic operators and nothing else. Only clamp is Zirric, composed from min and max.
• paths is entirely Go, wrapping the standard path package so that behaviour matches an established implementation rather than a hand-rolled one.
• fs splits evenly. The filesystem operations are Go, wrapping billy so that an in-memory filesystem and the host’s own are the same code; walk, glob, copy, withFile and the string helpers are Zirric, composed from those operations.
• random follows the same split: the generators are Go, everything derived from them is Zirric. Two of the three are bound by os rather than here, since they reach the machine.
• time is Go, but unusually its arithmetic lives in the VM rather than the module, since its three types are primitives rather than values built on top of one.
• clock is entirely Zirric. A clock is a closure over a reading, so the test clocks and the host’s alike are built in the language; os supplies only the two raw readings.
• reflect is Go only where it must be. Enumerating a module’s members needs access to the module value’s exports, describing a type’s fields needs what the compiler recorded about the declaration, and reaching a package’s modules needs the compiler itself; everything built on top of those — member, hasMember, typeOf, modulesWhere, modulesExcept — is Zirric, as are the Field and TypeRef types the Go side fills in. construct and fieldValues are the pair that takes a data value apart and puts one back together, which nothing else in the language can do for a type known only at runtime; ZE-021 is what needs them.

Characters versus bytes

len counts bytes, and indexing a String yields a Byte. Everything in strings that takes or returns a position instead counts characters, so count, charAt, slice, range, firstIndexOf and lastIndexOf all operate on Unicode code points. For "café", len is 5 while strings.count is 4.

Absence is an Option

Any operation that can fail to find something returns Option rather than a sentinel: arrays.first, arrays.last, bytes.firstIndexOf, strings.lastIndexOf, ranges.intersect and errors.join all follow this. The -1 returned by the underlying extern fn search primitives is converted at the Zirric boundary and never surfaces to callers.

A failure with something to say about itself is an Err instead, so time.parse, paths.match and strings.unquote return Result. The distinction is whether the caller learns anything from the reason: nothing follows from a needle simply not being there, while text that is not a quoted literal fails for a reason worth reporting.

Laziness in fun

fun.map, fun.filter, fun.flatMap, fun.take, fun.skip and fun.zip accept any @Iterable and return a lazy @Iterable. Nothing runs until the result is actually iterated, so the operations compose without building an intermediate array per step, and they terminate against infinite sources: fun.take(counter, 3) stops after three elements rather than hanging.

Laziness is carried by a single unexported _lazy type that wraps one step closure and implements @Iterable. Only one type is needed because the operations differ in what their step closure does, not in the shape of the protocol — so any future operation is another function returning a _lazy, with no new type.

fun.reduce is the exception and consumes its input eagerly, since folding to a single value has to visit every element. fun.zip is lazy in its first argument only; its second is buffered up front, because pairing by position needs random access that the push-based @Iterable protocol cannot provide incrementally for two independent sources.

The eager [Any]-specific equivalents in arrays are kept alongside these. They return arrays directly, which is what is wanted when the input is already an array and the result is about to be indexed or measured.

Time is three primitives, not one number

Duration, Instant and Timestamp are extern type declarations, which makes them primitives like Int or Binary rather than records wrapping one. Three things follow, and no other shape gives all three.

They get operators. Spans add, subtract and scale with +, - and *, which matters because this language has no operator overloading: a record would have turned every one of those into a function call, and durations are the most arithmetic-heavy thing here.

They print as themselves. A duration reads as 1.5s rather than 1500000000, both when inspected and when concatenated into text.

They carry a unit the VM enforces. The accepted combinations are the ones that mean something — a span plus a span, a span shifting a point, two points of the same kind differing by a span — and everything else is reported rather than coerced:

The last row is the reason Instant and Timestamp are separate at all. A monotonic reading and a wall-clock one are both a count of nanoseconds, so as plain numbers they would mix silently, and measuring elapsed time with a clock that can jump backwards is a bug that survives every test until the day the host is corrected.

time itself never reads a clock. Reading one is clock’s job, which is what leaves every function here pure and testable.

Randomness is a value too

random repeats the shape fs uses: a Source is a value, and the constructors differ only in where the bits come from. Code written against a Source neither knows nor cares which it was handed.

That is what makes randomness testable. random.seeded(n) produces the same sequence every run, so a function that shuffles, samples or picks can be tested for its actual behaviour rather than merely for not crashing — pass a seeded source and the outcome is fixed. fs.memory() does the same job for files.

The split between os.fastRandom and os.strongRandom is deliberate and named rather than implied. fastRandom is seeded from the clock and is a pseudo-random generator; strongRandom draws from the operating system’s cryptographic generator. Because both are a Source, moving from one to the other is a one-line change at the point of construction, and nothing downstream is rewritten.

Both live in os for the same reason the real clock does: they are the machine answering, and os is where the machine enters. What is left in random is seeded and the operations, none of which can surprise a test.

The capabilities are split the same way, and for a sharper reason than tidiness. A single HasRandom would be ambiguous exactly where the difference matters: code generating a token would accept whatever the environment offered, including a seeded generator. @HasStrongRandom and @HasFastRandom make the requirement part of the signature, so an environment providing only reproducible randomness cannot satisfy code that needs secrecy. A test environment can offer the fast capability alone and will be rejected, at the point of use, by anything asking for the strong one.

A filesystem is a value

fs describes a filesystem as a value rather than as a set of free functions, so a program can be handed one and never learn which it received. fs.memory() is a complete filesystem that touches no disk, which is what makes code that reads and writes files testable without fixtures or cleanup — every test in fs/_t runs against one.

Both implementations wrap billy, already a dependency of the package manager, so the in-memory and host filesystems are the same code over a different backing store rather than two implementations that drift.

cd returns a filesystem rooted at a subdirectory, which is how a program gives away a confined view of its own. It replaces what would otherwise be a separate os.dir.

Permissions are deliberately absent. billy puts them behind a capability its in-memory filesystem does not implement, so exposing them would mean an API that works on one filesystem and fails on another.

Streams are an attribute, not a type

io.Reader and io.Writer began as data types, which meant only io could produce a stream: anything wanting to accept one had to be handed a value of that exact type. As attributes they work the way @Countable and @Iterable already do — a program annotates its own type and it is a writer, accepted by fmt.fprint, the TAP reporter, or anything else asking for @Writer.

ReadStream and WriteStream remain as the types the host hands out, each holding a single function the runtime supplies. They are ordinary types carrying the attribute, with no privileged status.

This is also what makes a file a stream. A file has to be closable, so it cannot be an io.Writer value; carrying @Writer costs it nothing.

Module reflection needs only enumeration

A module is already a first-class value: import tests binds a prelude.Module, and a function value already answers name and arity and carries the attributes of its own declaration. Matching those attributes is ordinary Zirric — switch decl { case is @tests.Test: } — so the only thing missing was a way to ask a module what it declares. reflect adds exactly that, and nothing else needs Go.

This is why tests.discover is written in Zirric rather than as an extern. It walks reflect.members, keeps the declarations carrying @Test, and reads @Skip, @Todo and @Comment off each one the same way any other code would.

Only public members are reported, since a module value holds exactly its exports. _-prefixed declarations are internal and stay invisible, which is what makes strings._firstIndexOf absent from reflect.memberNames(strings) while strings.firstIndexOf is present.

A field is a value that carries its attributes

Reflecting a type is not enumeration, because a field is not a value the program can otherwise hold: Person has a field called name, but nothing in the language hands you that field. reflect.fieldsOf therefore builds one, and the important part is that the Field it builds carries the attributes written on the declaration as its own. That makes reading them ordinary Zirric — coding.Name(field).text and field is @coding.Ignore are the same expressions one would write against a function declaration — rather than a second, parallel way of asking about attributes.

A field’s declared type is kept structurally rather than as text, so [String: Person] arrives as a DictType of two NamedTypes and a decoder can walk it instead of parsing it. Each NamedType carries the type its name resolved to, which is the same value an is check compares against, so reflection can recurse into a nested data type without going back through a name.

Type hints are not enforced at runtime, which a TypeRef does not change: it says what a declaration promised, not what a value turned out to be.

Loading a package is deliberate

reflect.packages reaches the modules a package declares, including ones no import mentions. Reaching them at all means compiling them, which happens when the module is bound rather than when it is called; reflect.packages is therefore a separate module from reflect, so that ordinary module reflection — the part tests depends on — costs nothing.

Two kinds of module are never offered. The entry module is excluded because the running program cannot be loaded a second time. A module whose unqualified name is already taken by a loaded module is excluded because it could never be imported anyway: a project carrying its own prelude directory would otherwise compile a second set of core types, and values made by one would fail @Countable lookups from the other.

Loading a module runs its top-level code, so the API has no way to ask for every module without saying so. modulesWhere tests each name before resolving it, and modulesExcept requires the exclusions to be named — modulesExcept(p, []) loads everything, but only when written that way on purpose.

Changes to the Standard Library

arrays

Operates on [Any] and returns arrays eagerly. For lazy equivalents over any @Iterable, see fun.

bytes

clock

Reading the time is a side effect, so it lives apart from time. The capabilities are split because a wall clock can jump: code measuring how long something took must not be handed one.

The test clocks are the point. stepping and steppingMonotonic make time pass exactly as much as a test says it does, with no waiting, which is what fs.memory() and random.seeded() do for their own domains.

None of this module is Go. A clock is a closure over a reading, so every clock here — including the host’s, which os builds from two raw readings — is ordinary Zirric.

dicts

Iteration order is unspecified, so reduce must be given an order-independent combine.

errors

The concrete types behind join and wrap are intentionally unexported, so the only guaranteed surface is @Error itself.

fmt

Removed: println. Moved to the separate scripts package.

fs

Every operation that can fail returns a Result; nothing here stops the program. A File carries both @io.Reader and @io.Writer, so it can be passed anywhere a stream is expected, including fmt.fprint.

Every FileSystem field also exists as a module function taking the filesystem first — readFile, writeFile, open, create, exists, remove, move, list, mkdirAll, cd, root — so that fs.readFile(disk, path) reads like the rest of the library rather than disk.readFile(path), and so a filesystem can be threaded through fun.pipe and friends. The field remains the thing a custom filesystem implements; the function is only a call onto it.

The module is split across fs.zirr for the types, operations.zirr for those delegations, and helpers.zirr for everything composed from them.

fun

map, filter, flatMap, take, skip and zip take any @Iterable and return a lazy @Iterable. reduce is eager.

io

Reader and Writer became attributes rather than data types, so being a stream is something any type can carry instead of something only io can hand out. A program can annotate its own buffer, and it works with fmt.fprint and the TAP reporter like any other writer.

ReadStream and WriteStream are what the host provides — the concrete types behind os.stdout and friends — but they hold no privileged position.

The capability attributes let a value declare which streams it provides, so code can require a capability instead of reaching for os directly.

math

Int and Float mix freely in arithmetic, where the VM promotes to Float, and this module follows the same instinct: abs, min, max and clamp return whichever type they were given, while the rounding family, sqrt and pow return Float because that is what they genuinely produce. toInt is the way back.

Float division is unguarded, so 1.0 / 0.0 and 0.0 / 0.0 yield infinity and NaN without any effort on the caller’s part. isInfinite and isNaN are how those are detected — NaN in particular compares unequal to everything, itself included, so no equality test can find it.

options

from accepts any value: None stays None, an existing Some is left as it is, and anything else is lifted into Some. Every other function normalizes through it, so all of them accept bare values too.

os

New here are fs, the two clocks and the two random sources; the standard streams and process accessors are unchanged, though stdout, stdin and stderr now yield values carrying @io.Writer or @io.Reader rather than values of a Writer/Reader type.

fastRandom and strongRandom build a random.Source the same way systemClock builds a clock.SystemClock: the type and everything done with it belong to the module named after the domain, while the constructor reaching the machine belongs here. random is then left with seeded alone, which is the one source a test can rely on.

paths

Pure text manipulation: nothing here touches a filesystem. Paths are slash-separated regardless of host, matching the filesystem abstraction rather than the machine the program runs on.

prelude

Removed: bytesFromString and binaryFromString, superseded by bytes.fromString and bytes.fromChar. The length fields on Array, Dict and String, superseded by len. with and pipe, moved to fun. Bool.toggle.

Changed: Dict.keys and String.chars became methods, keys() and chars(). Func gained a name field.

random

A Source is a value, so code takes one rather than reaching for a generator. Each field has a matching module function taking the source first.

seeded is the only constructor here. The two drawing on the host are os.fastRandom and os.strongRandom, so that nothing in this module reaches the machine and a test is never one call away from real randomness.

ranges

reflect

Reports only a module’s public members, in name order. Enumeration is the one thing that needs Go; the values it returns already answer name and arity and carry their own attributes.

Types are reflected the same way. fieldsOf returns values, so the attributes written on a field are read off it exactly as off any other declaration.

reflect.packages

Reaching a package’s modules compiles them, so this is a separate module: importing reflect alone carries none of that cost. Loading a module runs its top-level code, which is why there is no function that loads every module without the caller saying so.

results

from normalizes any @AnyResult through its toResult, so every function accepts any result-shaped type, not just Result.

scripts

Conveniences that assume a process and its standard streams, so that fmt itself stays free of os.

scripts is the one module in this proposal that does not ship with the standard library. It lives in its own package, added to a Cavefile like any other dependency, because nothing else in the library needs it and a program that never prints should not carry it.

The same idea extends to files: these operate on os.fs(), so a script need not thread a filesystem through itself. Code that wants to stay testable takes an fs.FileSystem instead.

The package is split by what each group wraps — fmt-scripts.zirr and fs-scripts.zirr — so that adding helpers for another module adds a file rather than lengthening one.

strings

Positions count characters, not bytes. Most of this module is Go: indexing has to decode UTF-8, and building a string from parts in Zirric would be quadratic.

time

Pure throughout: nothing here reads a clock. The three types are primitives, so their arithmetic is the VM’s rather than this module’s — see Time is three primitives.

Time zones and calendar arithmetic such as “one month later” are deliberately absent: both need a zone database and rules that vary by locale, which is a larger commitment than a first version should make. Everything here is UTC.

tests

A test is a function annotated with @Test returning a Result. exec executes a list of TestCase and reports progress through an event callback, which keeps reporting independent of execution.

tests is only this vocabulary — declarations, no functions and no imports. Running and finding tests both live in tests.runner, so depending on the attributes and types drags in neither reflection nor a reporter.

tests.runner

Finds and runs tests. Kept apart from tests because it reaches for reflect, reflect.packages, os and a reporter, none of which writing a test requires.

exec is the primitive: it runs exactly the cases it is given, and neither discovers nor reports anything. @Only is applied there rather than during discovery, so restricting a run behaves the same however the cases were collected. Discovery loads only the modules its predicate accepts, so the rest never run. runT is the convention this repository follows and matches any module name ending in _t, which covers both a foo/_t submodule and a sibling module named foo_t; repositories that organize tests differently use runWhere instead. Both report TAP to stdout and exit non-zero when a test failed, so a task or CI step fails with the suite.

A discovered case is named <module>.<function>, so tests sharing a function name across modules stay distinguishable in the report.

tests.assert

Every assertion returns a Result, so a test body is an expression rather than a sequence of statements.

tests.tap

Deferred

Running other programs is left out. A useful interface needs to write to a program’s input while reading its output, and to run more than one at a time, neither of which is expressible while the language is single-threaded: anything built now would buffer everything and deadlock on a pipeline that fills a pipe. The shape it should take — a Processes value, a capability, and a separation between a program that fails and a program that will not start — survives in this proposal’s history rather than in code, to be revisited once concurrency exists.

Time zones and calendar arithmetic belong to time and are described there.

Alternatives Considered

Methods on the prelude types instead of modules. Helpers could have been added directly to Array, String and friends as fields on their extern type. That keeps calls shorter, but every helper would then have to be an extern fn implemented in Go, even the ones expressible in Zirric, and the prelude would grow without bound as the library does. Separate modules keep prelude to the types themselves and let most helpers be ordinary Zirric.

Splitting by type rather than by domain. bytes and binaries, or chars and strings, would mirror the types exactly. It was rejected because the boundary is arbitrary in practice: comparing two Binary values or searching a String for a Char belongs equally to both halves, so every new helper would need a placement decision that carries no information.

Overloading instead of Like unions. Accepting several types could have been expressed as overloaded declarations. A union plus a from normalizer was chosen instead because it needs no new language feature, it names the accepted set in one place that users can read, and it makes the normalization explicit rather than hidden in dispatch.

Eager fun operations returning arrays. fun.map and friends could have returned [Any] like their arrays counterparts. Laziness was chosen because these accept any @Iterable, including infinite ones, where an eager implementation cannot terminate, and because chained operations would otherwise allocate an intermediate array per step. The eager array versions remain available in arrays for the common case where the input is already an array.

Type reflection instead of module reflection. An earlier sketch (future/reflect/stub.zirr) described reflecting over types and fields — typeOf, fieldsOf, Field, Attribute. It is left as a sketch because nothing consumes it, while enumerating declarations is what test discovery and documentation generation both need. The two are independent axes and can be added separately.

A lazy type per operation. MapSeq, FilterSeq and so on would mirror the way some languages model lazy pipelines. A single _lazy wrapping one step closure was chosen because the operations differ only in that closure, so per-operation types would add declarations without adding capability, and each new operation would require another type.

Acknowledgements

The module split by domain and the naming of bytes, strings and errors follow Go’s standard library. The Result and Option helper sets — map, mapErr, flatMap, catch — follow Rust’s Result and Option combinators, and Swift’s Optional informs the Some-preserving behaviour of options.from. The lazy sequence operations in fun are modelled on Rust’s iterator adapters and Swift’s LazySequence, both of which likewise keep an eager collection-level API alongside the lazy one.