Native Memory Management in Clorch: A Practical Guide

Why tensors live outside the JVM heap, when to use with-torch scopes, how to retain or release pointers, and how to prevent memory leaks in hot loops.

Clorch tensors are JavaCPP wrappers around LibTorch objects allocated entirely outside the JVM heap. Each wrapper is a small Java object, but it may own hundreds of megabytes — or several gigabytes — of native CPU RAM or CUDA VRAM. Getting memory management right is the single most operationally important topic in Clorch, and this page covers it completely.

---

JVM GC and Native Memory

The JVM garbage collector can eventually reclaim an unreachable Clorch tensor because JavaCPP attaches a native deallocator to every wrapper object. When the GC finalises the Java wrapper, LibTorch frees the underlying allocation. However, the GC has no visibility into:

• Native heap pressure — a [4096 4096] float32 tensor occupies 64 MB of native RAM, but the JVM sees a tiny Java object.
• CUDA VRAM pressure — GPU memory exhaustion does not trigger JVM collection.
• Timing — finalisation is not guaranteed to run before the next allocation, especially under steady allocation pressure.

For small scripts and occasional operations this is acceptable. For any repeated loop that produces tensors — training batches, generation steps, MCMC draws — you need deterministic scopes.

Running a training loop without with-torch and relying on GC to release intermediates is the most common source of out-of-memory errors reported by Clorch users. CUDA OOM errors in particular give no opportunity for a recovery path.

---

When to Use with-torch

The table below maps workload patterns to the right strategy:

Workload	Recommendation
Small REPL expression or short script	GC is usually sufficient
Long-lived model, optimizer, or dataset	Keep it outside iteration scopes
Training or inference batch loop	Use one with-torch per iteration
Autoregressive generation or MCMC loop	Use one with-torch per step
Large CPU tensors	Use with-torch around temporary computation
CUDA workloads	Strongly prefer deterministic scopes
Interactive REPL session with many expressions	Use start-session! and stop-session!

The key insight is granularity: with-torch should wrap the smallest repeated unit that creates temporary tensors, not every individual operation.

---

Canonical Training Loop

Create the model and optimizer once before the loop. Scope only batch-local outputs, losses, and intermediates. Finish each scope with a JVM scalar or nil so no tensor escapes:

(require '[clorch.torch          :as t]
         '[clorch.nn             :as nn]
         '[clorch.nn.functional  :as F]
         '[clorch.autograd       :as autograd]
         '[clorch.optim          :as optim])

(let [model     (create-model)                      ;; lives outside scope
      optimizer (optim/adam (nn/parameters model))] ;; lives outside scope
  (doseq [{:keys [data target]} dataloader]
    (let [loss-value
          (t/with-torch                              ;; one scope per batch
            (optim/zero-grad optimizer)
            (let [prediction (nn/forward model data)
                  loss       (F/cross-entropy prediction target)]
              (autograd/backward loss)
              (optim/step optimizer)
              (t/item-float loss)))]                 ;; return JVM Float, not Tensor
      (println "Loss:" loss-value))))

The model and optimizer are created once and never owned by any with-torch scope — they are long-lived JVM references managed by the GC. Batch intermediates (prediction, loss) are created inside the scope, and item-float extracts a plain JVM Float as the final result, so the scope releases all native allocations when the block exits.

See the complete runnable examples pytorch_basics_tutorial.clj and synthetic.clj for end-to-end training loops using this pattern.

---

How with-torch Works

(t/with-torch
  (let [a (t/randn [1000 1000])
        b (t/randn [1000 1000])]
    (t/matmul a b)))

with-torch opens a JavaCPP PointerScope. Every native pointer allocated while the scope is open is registered with that scope. Before the scope closes, Clorch calls retain! recursively on the final result — the value of the last expression — to remove it from the scope’s ownership. When the scope closes, every remaining registered pointer is immediately deallocated. The final result is returned to the caller and is now managed by the JVM GC.

Returning maps and collections works too. retain! traverses map values and collection elements, so returning a structured result is safe:

(t/with-torch
  {:prediction prediction
   :attention   attention})
;; Both tensors are retained; all other intermediates are released.

The “accidentally retaining” anti-pattern

This loop retains one tensor per iteration and then silently discards it:

;; BAD: with-torch returns and retains compute-loss's result,
;; but doseq throws it away — native memory grows indefinitely.
(doseq [batch dataloader]
  (t/with-torch
    (compute-loss batch)))

Fix it by returning a JVM value or nil:

;; GOOD: scope returns nil → nothing is retained → all memory released.
(doseq [batch dataloader]
  (t/with-torch
    (let [loss (compute-loss batch)]
      (println (t/item-float loss))
      nil)))

---

Explicit Retention

retain! and rescue-pointers!

When a tensor needs to escape through state that with-torch cannot traverse — an atom, a delay, a closure, a cache, or any arbitrary Java object — call retain! before the scope closes:

(def saved (atom nil))

(t/with-torch
  (let [x (t/randn [5])]
    (t/retain! x)        ;; remove x from scope ownership
    (reset! saved x)
    nil))                ;; scope closes; x survives because it was retained

rescue-pointers! is an alias for retain! — use whichever name reads more clearly in context.

Simpler alternative: return from with-torch

When possible, just return the tensor as the with-torch result:

(reset! saved
        (t/with-torch
          (t/randn [5])))
;; No explicit retain! needed — with-torch retains the returned value.

This is equivalent and less error-prone than manual retain!.

---

Manual Release

release! immediately deallocates a pointer without waiting for the GC or a scope boundary. It recursively handles maps and collections:

(def x (t/randn [100 100]))

;; ... use x ...

(t/release! x)
;; x's native allocation is gone. x must not be used again.

After release!, the wrapper object is invalid. Any further operation on x — including reading its dtype or size — will crash with a native access error. All Clojure references to the same pointer become invalid simultaneously. Prefer lexical with-torch scopes unless you have clear, unambiguous ownership.

---

Interactive Session Scopes

For REPL development, creating a with-torch around every expression is impractical. Instead, open a long-lived session scope for the current thread:

(t/start-session!)

(try
  ;; All tensor allocations here attach to the session scope.
  ;; You can experiment freely without worrying about individual lifetimes.
  (do-repl-experiments)
  (finally
    ;; Releases all tensors created in this session.
    (t/stop-session!)))

Starting a new session automatically closes any existing session on the same thread. Session scopes are thread-local: worker threads and core.async go-blocks run on different threads and need their own scopes.

---

Forcing a GC Pass

gc! calls System/gc and System/runFinalization to prod the JVM into running pending finalizers:

(t/gc!)

gc! is intended for interactive diagnostics and recovery, not for steady-state memory management. The JVM may ignore explicit GC requests depending on GC configuration. It cannot release tensors that are still reachable from live Clojure references. Do not rely on gc! inside training loops.

---

Lifecycle API Reference

Function	Effect
with-torch	Opens a native pointer scope; releases all block-local pointers except those reachable from the final result
retain!	Removes a pointer (or all pointers in a collection/map) from its current scope, handing ownership to the GC
rescue-pointers!	Alias for retain!
release!	Immediately and unconditionally deallocates owned pointers; pointer is invalid after this call
start-session!	Opens a long-lived thread-local pointer scope for interactive REPL use
stop-session!	Closes the current thread’s interactive scope and releases all its pointers
gc!	Requests JVM GC and finalisation; diagnostic use only

---

Rules of Thumb

1. Keep model, optimizer, and all intentionally long-lived tensors outside iteration scopes.
2. Use one with-torch per allocating batch, generation step, or sampler step — not one per individual operation.
3. End a scope with a JVM scalar or nil unless a tensor must escape; this is the simplest way to guarantee no accidental retention.
4. Use retain! when a tensor must escape through an atom, closure, cache, or other side-effect that with-torch cannot discover.
5. Use release! only when ownership is clear and unambiguous — prefer scopes.
6. Do not depend on gc! for steady-state memory bounds; it is a diagnostic tool.
7. In REPL sessions, use start-session! / stop-session! rather than sprinkling with-torch around every expression.