Loopholing Discrete Diffusion Models

This repository provides the official PyTorch implementation for Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall.

Loopholing is deterministic latent propagation for discrete diffusion language models. It bypasses the sampling wall by carrying pre-sampling distributional context across denoising steps, instead of discarding it after categorical sampling collapses predictions into one-hot tokens.

By preserving this richer latent information, LDDMs stabilize denoising, reduce idle steps and oscillations, and improve non-autoregressive text generation. The latent pathway is trained efficiently with a self-conditioning objective, avoiding expensive trajectory unrolling.

Code Structure

├── main.py              # Entry point (train / ppl_eval / sample_eval)
├── algo.py              # All algorithm implementations
├── trainer_base.py      # Base classes (Diffusion, AbsorbingState, UniformState)
├── models/
│   └── dit.py           # Diffusion Transformer backbone
├── dataloader.py        # Data loading and tokenization
├── metrics.py           # NLL, Perplexity, Generative Perplexity, etc.
├── lm_eval_harness.py   # Downstream task evaluation (LM Eval Harness)
├── configs/             # Hydra configs (algo, data, model, noise, strategy)
└── scripts/             # Training and evaluation shell scripts

Algorithm hierarchy:

• MDLM → LDDM_M (masked diffusion + loophole)
• UDLM → LDDM_U (uniform diffusion + loophole)
• Baselines: SEDD, D3PM, AR

Pretrained Checkpoints

Pretrained checkpoints are available on Google Drive.

Installation

conda create -n lddm python=3.12
conda activate lddm
conda install nvidia/label/cuda-12.4.0::cuda-toolkit
pip install -r requirements.txt
pip install flash_attn==2.7.4.post1

Training

Training scripts are available in scripts/train/:

• LM1B: train_lm1b_lddm_m.sh, train_lm1b_lddm_u.sh
• OpenWebText: train_owt_lddm_m.sh, train_owt_lddm_u.sh

Example:

bash scripts/train/train_lm1b_lddm_m.sh

Evaluation

Evaluation scripts are in scripts/eval/:

• Perplexity: eval_lm1b_lddm_m.sh, eval_owt_lddm_m.sh
• Generation: gen_owt_lddm_m.sh

Downstream task scripts are in scripts/lm_eval/:

• Downstream: lm_eval.sh

Example:

bash scripts/eval/eval_owt_lddm_m.sh

Baselines

Baseline implementations (SEDD, MDLM, UDLM, D3PM, AR) are also included:

• Training: scripts/train/train_*_{sedd,mdlm,udlm}.sh
• Evaluation: scripts/eval/eval_*_{sedd,mdlm,udlm}.sh

Citation

@article{jo2025loopholing,
  title={Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall},
  author={Jo, Mingyu and Yoon, Jaesik and Deschenaux, Justin and Gulcehre, Caglar and Ahn, Sungjin},
  journal={arXiv preprint arXiv:2510.19304},
  year={2025}
}

Acknowledgements

This repository is based on DUO.