> ## Documentation Index
> Fetch the complete documentation index at: https://antlobach-clorch-182a83cb.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Clorch Example Programs: A Guided Code Walkthrough

> Hands-on walkthroughs of every example shipped with Clorch, from a minimal defmodel to distributed multi-GPU DDP training and Bayesian MCMC inference.

Clorch ships a set of self-contained example files that cover the full range of the library — from a ten-line `defmodel` to a Llama-3-style chat loop with KV caching and a multi-GPU distributed training harness with AMP and gradient accumulation. Each example is a runnable Clojure namespace; load it in a REPL or evaluate it top-to-bottom. The snippets below are taken directly from the source files.

<CardGroup cols={2}>
  <Card title="Simple Model" icon="cube" href="#simple-model">
    Minimal `defmodel` with one training step
  </Card>

  <Card title="PyTorch Basics Tutorial" icon="graduation-cap" href="#pytorch-basics-tutorial">
    Tensors, datasets, optimization, and model save/load
  </Card>

  <Card title="Autograd Tutorial" icon="arrows-spin" href="#autograd-tutorial">
    Gradient computation and computation-graph behavior
  </Card>

  <Card title="Synthetic Training" icon="chart-line" href="#synthetic-training">
    End-to-end training loop on generated data
  </Card>

  <Card title="Modern Llama" icon="robot" href="#modern-llama">
    RoPE, GQA, SwiGLU, and incremental KV caching
  </Card>

  <Card title="NanoChat" icon="comments" href="#nanochat">
    Compact Llama-style training, checkpointing, and chat generation
  </Card>

  <Card title="Distributed CUDA Training" icon="server" href="#distributed-cuda-training">
    NCCL workers, DDP, AMP, and gradient accumulation
  </Card>

  <Card title="Bayesian Linear Regression MCMC" icon="chart-scatter" href="#bayesian-linear-regression-mcmc">
    Posterior sampling with Metropolis-Hastings
  </Card>

  <Card title="Einsum eDSL" icon="sigma" href="#einsum-edsl">
    Declarative tensor contractions with `ein`
  </Card>
</CardGroup>

***

## Simple Model

**Source:** [`examples/simple.clj`](https://github.com/antlobach/clorch/blob/main/examples/simple.clj)

The simplest possible end-to-end demonstration of Clorch. It defines a two-layer MLP using the `defmodel` macro, constructs a fake batch, runs a forward pass, computes MSE loss, and performs one Adam optimizer step — all inside a `with-torch` scope to manage native memory correctly.

`defmodel` accepts constructor arguments, a binding vector of registered sub-modules, and a `forward` form. Registered fields participate automatically in `nn/parameters`, `nn/to`, and state dictionaries.

```clojure theme={null}
(ns simple
  (:require [clorch.torch :as t]
            [clorch.nn :as nn]
            [clorch.nn.functional :as F]
            [clorch.optim :as optim]
            [clorch.autograd :as autograd]))

(nn/defmodel MyModel [in out]
  [l1 (nn/linear in 16)
   l2 (nn/linear 16 out)]
  (forward [x]
           (let [h (nn/forward l1 x)
                 h (F/relu h)]
             (nn/forward l2 h))))

(t/with-torch
  (let [model (MyModel 10 3)
        x     (t/randn [4 10])
        y     (t/randn [4 3])
        opt   (optim/adam (nn/parameters model) :lr 1e-3)]
    (optim/zero-grad opt)
    (let [pred (nn/forward model x)
          loss (F/mse-loss pred y)]
      (autograd/backward loss)
      (optim/step opt))
    model))
```

***

## PyTorch Basics Tutorial

**Source:** [`examples/pytorch_basics_tutorial.clj`](https://github.com/antlobach/clorch/blob/main/examples/pytorch_basics_tutorial.clj)

A direct Clojure port of the official [PyTorch "Learn the Basics" tutorial](https://docs.pytorch.org/tutorials/beginner/basics/intro.html). It demonstrates tensor creation, a custom dataset backed by the `data/dataset` protocol, a multi-layer perceptron trained over multiple epochs with SGD and cross-entropy loss, and model serialization via `torch/save` and `torch/load`.

The training loop pattern — `zero-grad` → `forward` → `loss` → `backward` → `step` — matches PyTorch exactly and is idiomatic for all Clorch training code.

```clojure theme={null}
(defn train-epoch! [dataloader model loss-fn optimizer]
  (doseq [[batch-idx {:keys [data target]}] (map-indexed vector dataloader)]
    (torch/with-torch
      (let [target (torch/reshape target [(torch/size data 0)])
            pred   (nn/forward model data)
            loss   (loss-fn pred target)]
        (optim/zero-grad optimizer)
        (autograd/backward loss)
        (optim/step optimizer)
        (when (= 0 (mod (inc batch-idx) 2))
          (printf "  Loss: %.6f  [%d/%d]\n"
                  (torch/item-float loss)
                  (* (inc batch-idx) (torch/size data 0))
                  (data/get-size (:dataset dataloader))))))))
```

The dataset is built using `data/dataset` with `:size` and `:get-item` callbacks, returning `{:data … :target …}` maps that the dataloader batches automatically:

```clojure theme={null}
(defn make-synthetic-fashion-dataloader []
  (let [n-samples 64
        dataset (data/dataset
                 :size    (fn [] n-samples)
                 :get-item (fn [_idx]
                              {:data   (torch/randn [1 28 28])
                               :target (torch/tensor (rand-int 10) {:dtype :int64})}))]
    (data/dataloader dataset :batch-size 16 :shuffle true)))
```

***

## Autograd Tutorial

**Source:** [`examples/autograd_tutorial.clj`](https://github.com/antlobach/clorch/blob/main/examples/autograd_tutorial.clj)

A port of Sebastian Raschka's [Automatic Differentiation Made Easy](https://sebastianraschka.com/teaching/pytorch-1h/#4-automatic-differentiation-made-easy) tutorial. It covers scalar and tensor gradients, calling `backward` to populate `.grad` fields, reading gradients with `autograd/grad`, and iterating over tensor slices with `torch/tseq`.

```clojure theme={null}
(torch/with-torch
  (let [x (torch/tensor 2.0 {:requires-grad true})
        y (torch/mul x x)]

    (println "1. Basic Autograd: y = x^2 at x=2")
    (torch/tprint x)
    (torch/tprint y)

    (autograd/backward y)

    (let [g (autograd/grad x)]
      (println "dy/dx (should be 2x = 4):")
      (torch/tprint g)))

  (println "2. Sequence iteration test (tseq):")
  (let [t (torch/reshape (torch/tensor (map float (range 6))) [3 2])]
    (doseq [slice (torch/tseq t)]
      (println "Slice shape:" (torch/size slice)
               "Data:" (torch/item-float (torch/ix slice 0))))))
```

<Tip>
  Wrap autograd examples in `with-torch` to ensure intermediate tensors are released. The `autograd/grad` call returns the accumulated `.grad` tensor — do not retain it outside the scope unless you call `torch/retain!` explicitly.
</Tip>

***

## Synthetic Training

**Source:** [`examples/synthetic.clj`](https://github.com/antlobach/clorch/blob/main/examples/synthetic.clj)

An end-to-end training demonstration on procedurally generated multi-class data. It creates Gaussian clusters in 10-dimensional space, wraps them in a `tensor-dataset`, runs a two-layer ReLU MLP with SGD and cross-entropy loss for several epochs, and prints the epoch-averaged loss. It also demonstrates explicit cleanup of native resources with `data/cleanup-data!`.

```clojure theme={null}
(def model
  (nn/sequential
   (nn/linear d 4)
   (nn/relu)
   (nn/linear 4 n-classes)))

(nn/train model true)
(doseq [epoch (range num-epochs)]
  (let [losses (atom [])]
    (doseq [{:keys [data target]} train-loader]
      (torch/with-torch
        (let [logits (nn/forward model data)
              loss   (F/cross-entropy logits target)]
          (optim/zero-grad optimizer)
          (autograd/backward loss)
          (optim/step optimizer)
          (swap! losses conj (torch/item-float loss)))))
    (printf "Epoch %d | Avg Loss: %.4f\n"
            (inc epoch)
            (/ (apply + @losses) (count @losses)))))
```

***

## Modern Llama

**Source:** [`examples/modern_llama.clj`](https://github.com/antlobach/clorch/blob/main/examples/modern_llama.clj)

Demonstrates a single Llama-style transformer block built from Clorch's LLM primitives: `nn/GroupedQueryAttention` for grouped-query attention (fewer K/V heads than Q heads), `nn/SwiGLU` for the feed-forward block, `nn/rmsnorm` for pre-norm, and `torch/precompute-rope-freqs` + `torch/apply-rope` for rotary position embeddings. The example shows both a regular forward pass and an incremental KV-cache forward pass where a new token is appended to a prefix.

```clojure theme={null}
(nn/defmodel ModernLlamaBlock [dim n-heads n-kv-heads context-len drop-rate]
  [sa   (nn/GroupedQueryAttention dim n-heads n-kv-heads context-len drop-rate)
   ffwd (nn/SwiGLU dim (* 4 dim))
   ln1  (nn/rmsnorm dim)
   ln2  (nn/rmsnorm dim)]
  (forward [input]
           (let [{:keys [x mask freqs kv-cache]} (if (map? input) input {:x input})
                 x (torch/add x (nn/forward sa {:x     (nn/forward ln1 x)
                                                :mask  mask
                                                :freqs freqs
                                                :kv-cache kv-cache}))
                 x (torch/add x (nn/forward ffwd (nn/forward ln2 x)))]
             x)))
```

The KV-cache forward pass shows how to pass an `atom` as `:kv-cache` and slice pre-computed RoPE frequencies to the exact token positions being processed:

```clojure theme={null}
(def cache  (atom {}))
(def x1     (torch/randn [1 10 dim]))
(def freqs1 (torch/precompute-rope-freqs (quot dim n-heads) 10))
(def out1   (nn/forward cache-block {:x x1 :freqs freqs1 :kv-cache cache}))

;; Append one new token; slice frequencies to position [10, 11)
(def all-freqs (torch/precompute-rope-freqs (quot dim n-heads) 11))
(def freqs2    [(torch/ix (first all-freqs)  [10 11])
                (torch/ix (second all-freqs) [10 11])])
(def out2 (nn/forward cache-block {:x (torch/randn [1 1 dim])
                                   :freqs freqs2 :kv-cache cache}))
```

***

## NanoChat

**Source:** [`examples/nanochat.clj`](https://github.com/antlobach/clorch/blob/main/examples/nanochat.clj)

A compact, single-device Llama-3-style chat demo inspired by [Karpathy's NanoChat](https://github.com/karpathy/nanochat). It tokenizes a text corpus with `jtokkit`, trains a small `Llama` model (2 layers, 4 attention heads, 1 K/V head, 128-dimensional embeddings) using Adam with gradient clipping, saves a checkpoint, and supports interactive chat with streaming token generation.

The `Llama` model is built with `defmodel` nesting `LlamaBlock` records:

```clojure theme={null}
(nn/defmodel Llama [vocab-size emb-dim context-len n-layers n-heads n-kv-heads drop-rate]
  [tok-emb (nn/embedding vocab-size emb-dim :_freeze false)
   blocks  (vec (repeatedly n-layers
                  #(LlamaBlock emb-dim n-heads n-kv-heads context-len drop-rate)))
   ln-f    (nn/rmsnorm emb-dim)
   output  (nn/linear emb-dim vocab-size)
   head-dim (quot emb-dim n-heads)]
  (forward [input]
           (let [{:keys [idx mask freqs caches]} (if (map? input) input {:idx input})
                 x (nn/forward tok-emb idx)
                 x (loop [i 0 curr-x x]
                     (if (= i (count blocks)) curr-x
                       (recur (inc i)
                              (nn/forward (nth blocks i)
                                          {:x curr-x :mask mask
                                           :freqs freqs
                                           :kv-cache (when caches (nth caches i))}))))
                 x (nn/forward ln-f x)]
             (nn/forward output x))))
```

The generation loop samples from `F/softmax` probabilities, maintains a sliding context window, and optionally retains KV-cache tensors across steps using `torch/retain!`:

```clojure theme={null}
(defn generate-stream [model idx max-new-tokens context-size
                       & {:keys [use-cache?] :or {use-cache? true}}]
  (let [caches (when use-cache?
                 (vec (repeatedly (count (:blocks model)) #(atom {}))))]
    (autograd/no-grad
     (loop [i 0 curr-idx idx]
       (if (>= i max-new-tokens) curr-idx
         (let [next-idx
               (torch/with-torch
                 (let [logits     (nn/forward model {:idx curr-idx :freqs freqs-step
                                                     :caches caches})
                       last-logits (torch/ix logits :_ -1 :_)
                       probs       (F/softmax last-logits -1)
                       next-token  (torch/multinomial probs 1)]
                   (print (decode [(long (torch/item-float next-token))]))
                   (torch/cat [curr-idx next-token] 1)))]
           (recur (inc i) next-idx)))))))
```

<Note>
  `nanochat` requires `the-verdict.txt` in the working directory for training. It falls back gracefully to untrained weights for the `chat` function if no checkpoint exists.
</Note>

***

## Distributed CUDA Training

**Source:** [`examples/distributed_training.clj`](https://github.com/antlobach/clorch/blob/main/examples/distributed_training.clj)

A production-pattern multi-GPU training example demonstrating NCCL process groups, synchronous `DistributedDataParallel`, `clorch.amp` autocast with both `:float16` (dynamic scaling) and `:bfloat16`, gradient accumulation with `ddp/no-sync`, and rank-zero checkpoint saving via `dist/save-checkpoint!`.

The `train-worker` function is the per-rank entrypoint. It is passed `rank`, `world-size`, `process-group`, and `args` by the launcher:

```clojure theme={null}
(defn train-worker
  [{:keys [rank world-size process-group args]}]
  (let [{:keys [epochs sample-count batch-size accumulation precision checkpoint-path]} args
        sampler  (data/distributed-sampler
                   sample-count {:num-replicas world-size :rank rank :seed 1337})
        model    (nn/to (nn/linear 4 1) :cuda)
        optimizer (optim/adamw (nn/parameters model) :lr 0.01)
        scaler    (amp/grad-scaler {:enabled? (= precision :float16)})]
    (with-open [parallel-model
                (ddp/distributed-data-parallel
                  model {:process-group process-group :bucket-cap-mb 25.0})]
      (dotimes [epoch epochs]
        (data/set-epoch! sampler epoch)
        (doseq [micro-batches (partition-all accumulation
                                (partition-all batch-size
                                  (data/sample-indices sampler)))]
          (optim/zero-grad optimizer)
          (doseq [[micro-index indices] (map-indexed vector micro-batches)]
            (let [final-micro? (= micro-index (dec (count micro-batches)))
                  train-micro! (fn []
                    (t/with-torch
                      (let [{:keys [input target]} (batch-tensors indices sample-count)
                            loss (amp/autocast {:device :cuda :dtype precision}
                                   (F/mse-loss (nn/forward parallel-model input) target))]
                        (amp/backward! scaler (scaled-loss loss (count micro-batches))))))]
              (if final-micro? (train-micro!) (ddp/no-sync (train-micro!)))))
          (ddp/optimizer-step! parallel-model optimizer {:scaler scaler}))))))
```

Launch workers from the coordinator with `run-local!`, passing device indices and training hyperparameters:

```clojure theme={null}
(defn run-local!
  "Launches one training worker JVM per CUDA device and waits for completion."
  [devices & [options]]
  (let [job (dist/launch!
             {:nproc-per-node (count devices)
              :devices devices
              :main 'distributed-training/train-worker
              :args (or options {})})]
    (dist/await-job! job)
    {:status (dist/job-status job)
     :logs (into {}
                 (map (fn [rank] [rank (dist/job-logs job rank)]))
                 (range (count devices)))}))
```

<Warning>
  Distributed training requires two or more NVIDIA GPUs, CUDA 13.1, cuDNN 9, and NCCL 2. Set `CLORCH_FORCE_GPU=1` and `LD_LIBRARY_PATH` before starting the JVM. See the [README](https://github.com/antlobach/clorch/blob/main/README.md#multi-gpu-requirements) for the full host setup checklist.
</Warning>

***

## Bayesian Linear Regression MCMC

**Source:** [`examples/bayesian_linear_regression_mcmc.clj`](https://github.com/antlobach/clorch/blob/main/examples/bayesian_linear_regression_mcmc.clj)

Demonstrates probabilistic inference using `clorch.distributions`. It generates 120 synthetic observations from a known linear model, defines a log-posterior combining Normal priors on weights and bias with a Gaussian likelihood, and samples the posterior using random-walk Metropolis-Hastings. Multiple independent chains are run, with R-hat convergence diagnostics computed across chains.

```clojure theme={null}
(defn log-posterior [model {:keys [w b log-sigma]}]
  (let [sigma    (Math/exp (double log-sigma))
        lp-prior (+ (reduce + (map (fn [wj]
                                     (t/item-float
                                       (dist/log-prob (dist/normal 0.0 5.0) wj)))
                                   w))
                    (t/item-float (dist/log-prob (dist/normal 0.0 5.0) b))
                    (t/item-float (dist/log-prob (dist/normal 0.0 1.0) log-sigma)))
        _        (set-model-params! model {:w w :b b})
        mu       (nn/forward model x-t)
        lp-like  (t/item-float
                   (t/sum (dist/log-prob (dist/normal mu sigma) y-t)))]
    (+ lp-prior lp-like)))
```

Each Metropolis-Hastings step proposes a new state by adding Gaussian jitter, computes `log-alpha`, and accepts or rejects based on a uniform draw:

```clojure theme={null}
(defn mh-step [model {:keys [state logp accepted total]}]
  (t/with-torch
    (let [cand      (propose state)
          cand-logp (log-posterior model cand)
          log-alpha (min 0.0 (- cand-logp logp))
          accept?   (< (Math/log (t/item-float
                                   (dist/sample (dist/uniform 0.0 1.0))))
                       log-alpha)]
      (if accept?
        {:state cand :logp cand-logp :accepted (inc accepted) :total (inc total)}
        {:state state :logp logp     :accepted accepted        :total (inc total)}))))
```

Posterior summaries (mean, standard deviation, 5th/95th percentiles) and R-hat diagnostics are saved to `examples/out/` as CSV and EDN.

***

## Einsum eDSL

**Source:** [`examples/einsum_edsl.clj`](https://github.com/antlobach/clorch/blob/main/examples/einsum_edsl.clj)

Shows how to use `clorch.einsum`'s `ein` macro — a declarative Clojure eDSL for tensor contractions. Index variables are declared with `declare`, then used directly in `ein` expressions without string notation. The macro supports matrix-vector products, outer products, traces, and scalar-scaled contractions.

```clojure theme={null}
(ns examples.einsum-edsl
  (:require [clorch.einsum :refer [ein]]
            [clorch.torch :as t]))

(declare i j)

;; Matrix-vector product: y_i = sum_j A_ij * x_j
(def matrix-a (t/tensor [[1.0 2.0 3.0]
                         [4.0 5.0 6.0]]))
(def matrix-x (t/tensor [10.0 20.0 30.0]))
(def matrix-y (ein [i] := (* (matrix-a i j) (matrix-x j))))
;; => [140.0 320.0]

;; Outer product: M_ij = x_i * y_j
(def outer-x       (t/tensor [1.0 2.0 3.0]))
(def outer-y       (t/tensor [4.0 5.0]))
(def outer-product (ein [i j] := (* (outer-x i) (outer-y j))))

;; Trace: scalar = sum_i A_ii
(def trace-a     (t/tensor [[1.0 2.0] [3.0 4.0]]))
(def trace-value (ein [] := (trace-a i i)))
;; => 5.0

;; Scaled contraction: y_i = 0.5 * sum_j A_ij * x_j
(def scaled-y (ein [i] := (* 0.5 (scaled-a i j) (scaled-x j))))
```

<Tip>
  The `ein` eDSL uses Clojure's symbolic dispatch — index variables like `i` and `j` must be `declare`d at the top of the namespace before they appear in `ein` expressions.
</Tip>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.