Simultaneous GPU and CPU Usage with Llama Models

[jinzurei]

Description

Current setup prevents using GPU and CPU simultaneously for Llama model inference, limiting performance. We need support for heterogeneous computing (multi-device inference) to distribute workloads across GPU (for parallel tasks) and CPU (for sequential or memory-intensive ops), but it's not supported in our implementation.

Motivation

Improve efficiency and speed for Ignis by leveraging both devices, reducing bottlenecks in model loading and inference.

Proposed Implementation

• Investigate libraries like Accelerate or custom CUDA/CPU scheduling.
• Modify inference pipeline to split tasks (e.g., GPU for matrix ops, CPU for preprocessing).
• Test with small Llama models first.

Tasks

• Research heterogeneous computing options for Llama.
• Prototype multi-device setup.
• Benchmark performance gains.

Labels

bug, help-wanted, gpu, performance