Loss Functions for Regression and Classification Tasks

Reference for Clorch loss functions — MSE, L1, Huber, cross-entropy, NLL, BCE — with autograd notes and the zero-grad → backward → step training pattern.

Loss functions measure the discrepancy between a model’s predictions and the ground-truth targets, producing a scalar tensor whose gradient drives parameter updates. All Clorch loss functions live in clorch.nn.functional and return regular LibTorch tensors that participate in the autograd graph.

(require '[clorch.nn.functional :as F])

---

Regression Losses

Mean Squared Error

The standard choice for continuous regression targets. Penalises large errors quadratically.

(F/mse-loss input target)

L1 Loss (Mean Absolute Error)

Sums absolute differences rather than squared differences. More robust to outliers than MSE.

(F/l1-loss input target)

Smooth L1 Loss (Huber Loss)

Combines the best of MSE and L1: behaves like MSE for small errors (smooth gradient near zero) and like L1 for large errors (bounded gradient, resistant to outliers). Commonly used in object detection.

(F/smooth-l1-loss input target)

When you are unsure whether your dataset contains outliers, prefer smooth-l1-loss over plain mse-loss. It converges reliably in both clean and noisy settings.

---

Classification Losses

Cross Entropy

Combines log-softmax and negative log-likelihood in a single numerically stable operation. Expects raw logits (before any softmax), not probabilities.

;; logits: [batch, num-classes]   targets: [batch] int64
(F/cross-entropy logits targets)

NLL Loss — Negative Log Likelihood

Expects log-probabilities as input (e.g. the output of F/log-softmax). Useful when you want to decouple the softmax step from the loss.

(F/nll-loss (F/log-softmax logits -1) targets)

Binary Cross Entropy

For binary classification where each output is an independent probability in (0, 1). Input must already be passed through a sigmoid.

;; input: probabilities in (0, 1) — apply sigmoid first
(F/bce-loss (F/sigmoid logits) target)

BCE with Logits

Numerically more stable than applying sigmoid then BCE separately. Accepts raw logits directly.

;; input: raw logits — sigmoid + BCE fused in one kernel
(F/bce-with-logits-loss logits target)

Always prefer F/bce-with-logits-loss over (F/bce-loss (F/sigmoid x) target). The fused version avoids floating-point overflow that can occur when exponentiating very large logits.

---

Usage in a Training Loop

Loss tensors retain the full autograd graph until you call autograd/backward. The canonical pattern is:

(require '[clorch.nn :as nn]
         '[clorch.nn.functional :as F]
         '[clorch.optim :as optim]
         '[clorch.autograd :as autograd]
         '[clorch.torch :as t])

(defn train-step [model optimizer x y]
  ;; 1. Clear gradients from the previous iteration
  (optim/zero-grad optimizer)
  ;; 2. Forward pass
  (let [pred (nn/forward model x)
        ;; 3. Compute loss
        loss (F/cross-entropy pred y)]
    ;; 4. Backpropagate
    (autograd/backward loss)
    ;; 5. Update parameters
    (optim/step optimizer)
    ;; 6. Extract a plain JVM float before the scope closes
    (t/item-float loss)))

Loss tensors — and all intermediate activations — hold native memory. Wrap each training iteration in t/with-torch (or an equivalent scope) so that non-root tensors are released after the optimizer step. See the Memory Management page for the canonical training loop pattern.

---

Autograd Graph and Retention

Every call to a loss function produces a tensor whose .requires_grad is true as long as at least one input tensor requires gradients. Calling autograd/backward on that tensor computes and accumulates gradients into every leaf parameter.

Multiple backward passes

By default LibTorch frees the intermediate computation graph after a single backward call. If you need to call backward more than once (e.g. for second-order gradients or gradient accumulation with manual graph retention), pass the :retain-graph true option:

(autograd/backward loss :retain-graph true)

Retaining the graph keeps all intermediate activations in native memory until you release them explicitly, so use this only when necessary.