docs: add troubleshooting guide, tensor shapes reference, and FAQ

[yurekami]

Summary

This PR adds three comprehensive documentation files to help users troubleshoot issues, understand tensor shapes, and find answers to common questions. These documents are based on analysis of 53+ open GitHub issues and address the most frequent pain points reported by the community.

Changes

• docs/TROUBLESHOOTING.md - GPU compatibility matrix, build error solutions, common runtime errors
• docs/TENSOR_SHAPES.md - Complete tensor shape reference for all major functions
• docs/FAQ.md - Answers to frequently asked questions from GitHub issues

Motivation

The FlashMLA repository has 53+ open issues, many of which are:

• Questions about GPU architecture support (SM90 vs SM100 vs SM120)
• Confusion about sparse vs dense attention modes
• Build errors related to CUDA versions
• Tensor shape mismatches
• Integration questions with vLLM/SGLang

This documentation directly addresses these recurring questions, which should:

1. Reduce maintainer support burden
2. Improve developer onboarding experience
3. Make the project more accessible to new users

Documentation Overview

TROUBLESHOOTING.md

• GPU compatibility matrix (SM90/SM100/SM120)
• Build error solutions by CUDA version
• Runtime error fixes (e.g., "Sparse BF16 MLA is not supported on SM90")
• Windows/ARM64 workarounds
• Links to relevant GitHub issues

TENSOR_SHAPES.md

• Shape notation legend
• Complete function signatures with shape annotations:
  • flash_mla_with_kvcache
  • get_mla_metadata
  • flash_mla_prefill
  • mha_fwd_kvcache
• ASCII diagrams for paged KV cache layout
• Common shape mismatch errors and fixes

FAQ.md

• 10 frequently asked questions with detailed answers
• Topics: paged mode, prefill stage, GPU support, sparse attention, framework integration, performance expectations, variable-length batching, FP8 quantization

Related Issues

This documentation addresses questions raised in:

• Sparse MLA support for DSv32 #101, No code for sm100 dense mla. #113, support for rtx 6000 sm120 #124 (GPU architecture support)
• metadata api compatibility issue #108, RuntimeError: Sparse BF16 MLA is not supported on SM90 #110, how to test sparse attention in test_flash_mla_decoding #116 (sparse attention and API)
• Built FlashMLA for Windows and Nvidia sm_120 (workstation/50s) cards #119, Build error on cuda13 arm64 #121, does flash_mla_with_kvcache work only in paged mode? #126 (builds and compatibility)

Test Plan

• Verified all markdown renders correctly on GitHub
• Cross-referenced with existing README to ensure consistency
• Validated technical accuracy against FlashMLA source code and issues

🤖 Generated with Claude Code