AITune Release v0.4.1

AITune Release v0.4.1

AITune v0.4.1 improves profiling accuracy, lowers tuning memory spikes, and cleans up performance validation configuration names.

Summary

This release focuses on profiling and validation fixes. Baseline performance now uses the recorded global batch size, throughput is computed from mean latency, and fixed-step sampling is the default after warmup.

It also reduces memory spikes in baseline profiling and Torch Inductor JIT with autocast. Optional stable-window profiling now has clearer bounds, warmer failure messages, and a less strict default CV threshold.

Key Dependencies

• Python =3.10,3.13
• PyTorch =2.7,2.11
• Torch-TensorRT 2,2.11
• TorchAO =0.13,0.17
• TensorRT =10.5
• NumPy =2.0.0
• Optional Dynamo integration via aitune[dynamo]

Breaking Changes

• Profiling and inspection threshold arguments were renamed to ratio-based names.
• enable_validate_against_baseline(False) was replaced by enable_performance_validation(False).

Fixes & Improvements

Profiling

• Uses the recorded global batch size for baseline performance profiling.
• Computes profiling throughput from mean latency.
• Defaults profiling measurements to fixed-step sampling after warmup.
• Excludes warmup samples from optional stable-window CV checks.
• Adds max-sample bounds and clearer failure errors for stable-window profiling.
• Relaxes the default stable-window CV threshold to 10%.
• Validates ratio and positive profiling configuration values.

Memory Use

• Removes a memory spike during baseline performance measurement.
• Removes a memory spike in the Torch Inductor JIT backend when autocast is active.

Backend Behavior

• Aligns Torch-TensorRT defaults.
• Synchronizes CUDA during JIT inspection so measurements are more accurate.

Known Issues

• Hardware-specific quantization paths such as NVFP4 require compatible NVIDIA hardware and installed backend support.
• GPU functional tests require the relevant backend dependencies and hardware; unit tests remain the default local validation path.
• PyTorch 2.11 is outside the supported range for this release.

AITune Release v0.4.0

AITune Release v0.4.0

AITune v0.4.0 expands backend coverage, improves JIT tuning for complex pipelines, and adds observability needed to understand tuning decisions.

Summary

This release adds ONNXRuntime and Torch Inductor AOT backends, extends quantization support for TorchAO and TensorRT, introduces Dynamo serving support, and improves JIT tuning control for complex pipelines such as diffusion workloads.

It also adds performance validation against Torch eager baselines, richer tuning telemetry, optional hardware metrics, refreshed documentation and examples.

Key Dependencies

• Python >=3.10,<3.13
• PyTorch >=2.7,<2.11
• Torch-TensorRT >2,<2.11
• TorchAO >=0.13,<0.17
• TensorRT >=10.5
• NumPy >=2.0.0
• Optional Dynamo integration via aitune[dynamo]

Breaking Changes

• TorchInductorBackend was renamed to TorchInductorJitBackend.
• HighestThroughputStrategy was renamed to MaxThroughputStrategy.
• NVTX_ENABLE was renamed to AITUNE_NVTX_EVENTS.
• aitune.torch.jit_config.backends was removed. Configure JIT backends through aitune.torch.jit_config.strategy, for example FirstWinsStrategy(backends=[...]).
• The previous public system-monitoring APIs were removed:
• SystemMonitor
• system_resource_monitor
• enable_gpu_memory_logging

Features & Improvements

Backend Coverage

• Added ONNXRuntime backend support with CUDA and TensorRT execution providers.
• Added Torch Inductor AOT backend support for saved compiled artifacts.
• Added TorchAO NVFP4DQ and MXFP8DQ quantization options, including filter support and hardware validation.
• Added TensorRT NVFP4 quantization support.
• Improved dynamic-shape handling across Torch Inductor, Torch-TensorRT, and ONNX export paths.

JIT Tuning

• Added deferred JIT tuning mode for pipelines where modules run variable numbers of times before the best tuning point is known.
• Added JIT tune strategy selection through aitune.torch.jit_config.strategy.
• Added package and module-class exclusions for JIT patching.
• Added Diffusers integration hooks for pipeline compatibility.
• Improved wrapped-descendant handling so JIT tuning restores patched modules correctly.

Tune Strategies

• Backends that are correct but slower than Torch eager can now be rejected automatically.
• Added per-module speedup reporting during tuning.
• Speedup summaries are visible at the default warning log level.

Observability

• Added tuning telemetry reports covering runs, modules, graphs, backend builds, throughput, selected backends, and failures.
• Added AITUNE_TUNING_DATA_PATH for choosing the telemetry output path.
• Added snapshot_tuning_data() for long-running processes.
• Added optional hardware metrics collection with AITUNE_HARDWARE_METRICS=1.
• Added AITUNE_HARDWARE_METRICS_PATH for hardware metrics output.
• Added NVTX annotation support through AITUNE_NVTX_EVENTS=1.

Deployment & Runtime

• Added Dynamo worker support for serving AITune-tuned models as Dynamo endpoints.
• Fixed checkpoint backend artifacts portability by storing relative artifact paths.
• Added disk-space checks before build and save operations.

Bug Fixes

• Fixed Torch-TensorRT AOT save failures for wrapt-decorated forwards.
• Fixed shared dynamic dimensions in Torch-TensorRT AOT.
• Fixed bounded dimension handling for Torch-TensorRT AOT export.
• Fixed Torch Inductor AOT dynamic-shape handling.
• Fixed JIT tuning state handling when wrappers are still in initial state.
• Preserved externally registered forward hooks across save and restore.
• Fixed UserDict traversal for inputs such as transformers.BatchEncoding.
• Fixed Blackwell-specific quantization configuration handling.
• Dropped the Torch-TensorRT enabled_precisions={float16} default so engines follow the model dtype.

Documentation

• Migrated the documentation preview to Fern.
• Added backend guides for ONNXRuntime and Torch Inductor AOT.
• Refreshed backend, tune strategy, observability, deployment, and JIT guides.
• Updated notebooks and examples for current imports and backend defaults.
• Added agent workflow guidance, contribution guidance, code of conduct, and security policy files.

Known Issues

• Hardware-specific quantization paths such as NVFP4 require compatible NVIDIA hardware and installed backend support.
• GPU functional tests require the relevant backend dependencies and hardware; unit tests remain the default local validation path.
• PyTorch 2.11 is outside the supported range for this release.

AITune Release v0.3.0

Summary

AITune is an open-source (Apache 2.0) inference toolkit, hosted under the ai-dynamo GitHub organization and distributed via PyPI. It is designed for tuning and deploying Deep Learning models on NVIDIA GPUs, significantly improving inference speed and efficiency across various AI workloads.

Major Features & Improvements

Tuning Modes

• Just-in-Time (JIT) Tuning: Zero-code model tuning and inspection controlled through a single import or environment flag. Tunes on the very first model call using only one sample, with automatic fallback to Torch Inductor when a backend cannot compile a module.
• Ahead-of-Time (AOT) Tuning: Low-code API for explicit model inspection, backend selection, and module-level tuning. Supports forward hooks for custom pre/post-processing logic around tuned modules.

Backend Support

• TensorRT: Multi-profile engines with auto-generated and user-provided profiles, CUDA graph capture, FP16/FP8/INT8 mixed precision via TensorRT Model Optimizer, and Dynamo-based ONNX export (torch.onnx.export(dynamo=True)) for improved graph fidelity.
• TorchInductor: Added support for static and dynamic HuggingFace models, broadening model compatibility beyond TensorRT workflows.

Model Compatibility

• Complex Inputs: Support for dataclasses, user-defined objects in module.forward() arguments, and lists/dicts within Torch module containers for more complete model analysis.
• LLM Support: Added KV cache support to enable tuning of autoregressive large language models.

Performance & Observability

• Memory Optimization: Reduced CPU/GPU memory usage during tuning by offloading inactive modules to the meta device, with optimized input/output metadata handling.
• Profiling: Extended metrics collection through NVTX annotations for Nsight Systems integration. Added configurable console output suppression with automatic log-to-file.

Documentation & Examples

• Documentation & Examples: Added comprehensive documentation and extended end-to-end examples across Computer Vision, Generative AI, Speech Recognition, and NLP workloads.

Bug Fixes

• Fixed dynamic shapes handling in TorchTensorRT AoT and TensorRT ONNX Dynamo export paths, calibration data creation for ModelOpt PTQ, bfloat16 precision in TensorRT, JIT cache directory collisions, and profiling for models without batching support.

Known Issues

• AITune currently only supports single-GPU configurations.
• Just-in-Time tuning does not support transformers>=5 due to @capture_outputs decorator.

Full Changelog

aitune/CHANGELOG.md