Zijie Tian c51a640a29 🐛 fix: remove torch.compile from add_rms_forward to avoid recompilation ...

The add_rms_forward method processes two input tensors (x and residual),
which causes torch.compile recompilation issues. Keep @torch.compile only
on rms_forward which processes a single input.

This prevents unnecessary recompilation overhead during inference.

Co-Authored-By: Claude <noreply@anthropic.com>

2026-01-14 07:02:02 +08:00

..

comm

[WIP] Added sgDMA operator for scatter kvcache communication.

2025-12-24 23:48:52 +08:00

debug

[refactor] Refactor the kvcache offload.

2026-01-04 19:37:03 +08:00

engine

♻️ refactor: chunked LayerNorm/QKV/MLP for 64k memory optimization

2026-01-14 07:01:57 +08:00

kvcache

[claudesquad] update from 'multi-request-2' on 13 Jan 26 02:01 CST

2026-01-13 02:01:07 +08:00

layers

🐛 fix: remove torch.compile from add_rms_forward to avoid recompilation

2026-01-14 07:02:02 +08:00

models

[tests] Added test_niah_standalone.py.

2026-01-12 00:16:37 +08:00

utils

[claudesquad] update from 'lw-offload-2' on 08 Jan 26 20:53 CST

2026-01-08 20:53:08 +08:00

__init__.py

better

2025-06-15 10:36:45 +08:00

config.py

[tests] Added test_niah_standalone.py.

2026-01-12 00:16:37 +08:00

llm.py

support tensor parallel

2025-06-15 01:31:24 +08:00

sampling_params.py

compile random sampling

2025-08-31 22:55:34 +08:00