> ## Documentation Index > Fetch the complete documentation index at: https://antlobach-clorch-182a83cb.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Native Memory Management in Clorch: A Practical Guide > Why tensors live outside the JVM heap, when to use with-torch scopes, how to retain or release pointers, and how to prevent memory leaks in hot loops. Clorch tensors are JavaCPP wrappers around LibTorch objects allocated entirely outside the JVM heap. Each wrapper is a small Java object, but it may own hundreds of megabytes — or several gigabytes — of native CPU RAM or CUDA VRAM. Getting memory management right is the single most operationally important topic in Clorch, and this page covers it completely. *** ## JVM GC and Native Memory The JVM garbage collector can eventually reclaim an unreachable Clorch tensor because JavaCPP attaches a native deallocator to every wrapper object. When the GC finalises the Java wrapper, LibTorch frees the underlying allocation. However, the GC has no visibility into: * **Native heap pressure** — a `[4096 4096]` float32 tensor occupies 64 MB of native RAM, but the JVM sees a tiny Java object. * **CUDA VRAM pressure** — GPU memory exhaustion does not trigger JVM collection. * **Timing** — finalisation is not guaranteed to run before the next allocation, especially under steady allocation pressure. For small scripts and occasional operations this is acceptable. For any repeated loop that produces tensors — training batches, generation steps, MCMC draws — you need deterministic scopes. Running a training loop without `with-torch` and relying on GC to release intermediates is the most common source of out-of-memory errors reported by Clorch users. CUDA OOM errors in particular give no opportunity for a recovery path. *** ## When to Use `with-torch` The table below maps workload patterns to the right strategy: | Workload | Recommendation | | - | - | | Small REPL expression or short script | GC is usually sufficient | | Long-lived model, optimizer, or dataset | Keep it **outside** iteration scopes | | Training or inference batch loop | Use one `with-torch` per iteration | | Autoregressive generation or MCMC loop | Use one `with-torch` per step | | Large CPU tensors | Use `with-torch` around temporary computation | | CUDA workloads | Strongly prefer deterministic scopes | | Interactive REPL session with many expressions | Use `start-session!` and `stop-session!` | The key insight is granularity: `with-torch` should wrap the smallest repeated unit that creates temporary tensors, not every individual operation. *** ## Canonical Training Loop Create the model and optimizer once before the loop. Scope only batch-local outputs, losses, and intermediates. Finish each scope with a JVM scalar or `nil` so no tensor escapes: ```clojure theme={null} (require '[clorch.torch :as t] '[clorch.nn :as nn] '[clorch.nn.functional :as F] '[clorch.autograd :as autograd] '[clorch.optim :as optim]) (let [model (create-model) ;; lives outside scope optimizer (optim/adam (nn/parameters model))] ;; lives outside scope (doseq [{:keys [data target]} dataloader] (let [loss-value (t/with-torch ;; one scope per batch (optim/zero-grad optimizer) (let [prediction (nn/forward model data) loss (F/cross-entropy prediction target)] (autograd/backward loss) (optim/step optimizer) (t/item-float loss)))] ;; return JVM Float, not Tensor (println "Loss:" loss-value)))) ``` The model and optimizer are created once and never owned by any `with-torch` scope — they are long-lived JVM references managed by the GC. Batch intermediates (`prediction`, `loss`) are created inside the scope, and `item-float` extracts a plain JVM `Float` as the final result, so the scope releases all native allocations when the block exits. See the complete runnable examples [`pytorch_basics_tutorial.clj`](https://github.com/antlobach/clorch/blob/main/examples/pytorch_basics_tutorial.clj) and [`synthetic.clj`](https://github.com/antlobach/clorch/blob/main/examples/synthetic.clj) for end-to-end training loops using this pattern. *** ## How `with-torch` Works ```clojure theme={null} (t/with-torch (let [a (t/randn [1000 1000]) b (t/randn [1000 1000])] (t/matmul a b))) ``` `with-torch` opens a JavaCPP `PointerScope`. Every native pointer allocated while the scope is open is registered with that scope. Before the scope closes, Clorch calls `retain!` recursively on the final result — the value of the last expression — to remove it from the scope's ownership. When the scope closes, every remaining registered pointer is immediately deallocated. The final result is returned to the caller and is now managed by the JVM GC. **Returning maps and collections works too.** `retain!` traverses map values and collection elements, so returning a structured result is safe: ```clojure theme={null} (t/with-torch {:prediction prediction :attention attention}) ;; Both tensors are retained; all other intermediates are released. ``` ### The "accidentally retaining" anti-pattern This loop retains one tensor per iteration and then silently discards it: ```clojure theme={null} ;; BAD: with-torch returns and retains compute-loss's result, ;; but doseq throws it away — native memory grows indefinitely. (doseq [batch dataloader] (t/with-torch (compute-loss batch))) ``` Fix it by returning a JVM value or `nil`: ```clojure theme={null} ;; GOOD: scope returns nil → nothing is retained → all memory released. (doseq [batch dataloader] (t/with-torch (let [loss (compute-loss batch)] (println (t/item-float loss)) nil))) ``` *** ## Explicit Retention ### `retain!` and `rescue-pointers!` When a tensor needs to escape through state that `with-torch` cannot traverse — an `atom`, a `delay`, a closure, a cache, or any arbitrary Java object — call `retain!` before the scope closes: ```clojure theme={null} (def saved (atom nil)) (t/with-torch (let [x (t/randn [5])] (t/retain! x) ;; remove x from scope ownership (reset! saved x) nil)) ;; scope closes; x survives because it was retained ``` `rescue-pointers!` is an alias for `retain!` — use whichever name reads more clearly in context. ### Simpler alternative: return from `with-torch` When possible, just return the tensor as the `with-torch` result: ```clojure theme={null} (reset! saved (t/with-torch (t/randn [5]))) ;; No explicit retain! needed — with-torch retains the returned value. ``` This is equivalent and less error-prone than manual `retain!`. *** ## Manual Release `release!` immediately deallocates a pointer without waiting for the GC or a scope boundary. It recursively handles maps and collections: ```clojure theme={null} (def x (t/randn [100 100])) ;; ... use x ... (t/release! x) ;; x's native allocation is gone. x must not be used again. ``` After `release!`, the wrapper object is invalid. Any further operation on `x` — including reading its dtype or size — will crash with a native access error. All Clojure references to the same pointer become invalid simultaneously. Prefer lexical `with-torch` scopes unless you have clear, unambiguous ownership. *** ## Interactive Session Scopes For REPL development, creating a `with-torch` around every expression is impractical. Instead, open a long-lived session scope for the current thread: ```clojure theme={null} (t/start-session!) (try ;; All tensor allocations here attach to the session scope. ;; You can experiment freely without worrying about individual lifetimes. (do-repl-experiments) (finally ;; Releases all tensors created in this session. (t/stop-session!))) ``` Starting a new session automatically closes any existing session on the same thread. Session scopes are **thread-local**: worker threads and core.async go-blocks run on different threads and need their own scopes. *** ## Forcing a GC Pass `gc!` calls `System/gc` and `System/runFinalization` to prod the JVM into running pending finalizers: ```clojure theme={null} (t/gc!) ``` `gc!` is intended for interactive diagnostics and recovery, not for steady-state memory management. The JVM may ignore explicit GC requests depending on GC configuration. It cannot release tensors that are still reachable from live Clojure references. Do not rely on `gc!` inside training loops. *** ## Lifecycle API Reference | Function | Effect | | - | - | | `with-torch` | Opens a native pointer scope; releases all block-local pointers except those reachable from the final result | | `retain!` | Removes a pointer (or all pointers in a collection/map) from its current scope, handing ownership to the GC | | `rescue-pointers!` | Alias for `retain!` | | `release!` | Immediately and unconditionally deallocates owned pointers; pointer is invalid after this call | | `start-session!` | Opens a long-lived thread-local pointer scope for interactive REPL use | | `stop-session!` | Closes the current thread's interactive scope and releases all its pointers | | `gc!` | Requests JVM GC and finalisation; diagnostic use only | *** ## Rules of Thumb 1. Keep model, optimizer, and all intentionally long-lived tensors **outside** iteration scopes. 2. Use **one `with-torch` per allocating batch, generation step, or sampler step** — not one per individual operation. 3. End a scope with a **JVM scalar or `nil`** unless a tensor must escape; this is the simplest way to guarantee no accidental retention. 4. Use `retain!` when a tensor must escape through an atom, closure, cache, or other side-effect that `with-torch` cannot discover. 5. Use `release!` only when ownership is clear and unambiguous — prefer scopes. 6. Do **not** depend on `gc!` for steady-state memory bounds; it is a diagnostic tool. 7. In REPL sessions, use `start-session!` / `stop-session!` rather than sprinkling `with-torch` around every expression. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.