Zijie Tian
39d12a0416
📈 feat: add MemoryObserver for GPU-CPU communication tracking
...
Implement MemoryObserver to track memory transfers between GPU and CPU:
- H2D (Host to Device): CPU → GPU transfers
- D2H (Device to Host): GPU → CPU transfers
- D2D (Device to Device): GPU buffer copies
- Supports prefill/decode phase separation
Integration points in offload_engine.py:
- load_to_slot_layer: H2D with is_prefill parameter
- offload_slot_layer_to_cpu, offload_prefill_buffer_async: D2H
- write_to_prefill_buffer, write_to_decode_buffer: D2D
- load_block_sample_from_cpu, load_block_full_from_cpu: H2D
Add bench_offload.py integration for memory stats printing.
Benchmark results (Llama-3.1-8B, 64K context):
- Full Policy: Prefill H2D 262.13 GB
- XAttention: Prefill H2D 386.62 GB (1.48x)
Generated with [Claude Code](https://claude.ai/code )
via [Happy](https://happy.engineering )
Co-Authored-By: Claude <noreply@anthropic.com >
Co-Authored-By: Happy <yesreply@happy.engineering >
2026-01-28 04:06:45 +08:00
Zijie Tian
c16bfcf40f
♻️ refactor: restructure Observer as base class with InferenceObserver
...
- Refactor Observer into base class with common enable/disable/reset interface
- Create InferenceObserver subclass for TTFT/TPOT metrics
- Fix TTFT calculation timing: compute after prefill completes instead of
at decode start (fixes max_tokens=1 returning TTFT=0)
- Integrate InferenceObserver into bench.py and bench_offload.py for
accurate internal timing metrics vs external wall-clock time
- Add get_summary() and print_summary() methods for structured output
Generated with [Claude Code](https://claude.ai/code )
via [Happy](https://happy.engineering )
Co-Authored-By: Claude <noreply@anthropic.com >
Co-Authored-By: Happy <yesreply@happy.engineering >
2026-01-28 03:15:33 +08:00
Zijie Tian
50520a6c3c
[fix] fixed request to request error.
2026-01-19 00:55:26 +08:00
Zijie Tian
e554d5482b
[refactor] Delete unnesscessory test, and refacrtor the offload prefix cache.
2026-01-05 20:31:42 +08:00
Zijie Tian
89f8020d38
[WIP] fixing attention compute error.
2025-12-30 00:31:48 +08:00
Zijie Tian
0b6f19242d
[feat] Added chunked prefill and kvcache offload mechenism.
2025-12-10 03:47:37 +08:00
Zijie Tian
204fe2b38f
[feat] Added metric into tqdm bar.
2025-12-10 00:52:13 +08:00
GeeeekExplorer
df99418f7d
simplify
2025-08-31 20:02:51 +08:00
xiaohajiayou
054aec852d
Fix: Division-by-Zero Risk and Typo
2025-06-24 02:02:33 +08:00
Xingkai Yu
4fc764f175
Merge pull request #22 from cheunglei/use_spawn
2025-06-17 23:53:59 +08:00
cheunglei
b5ace32982
use spawn
2025-06-17 23:49:15 +08:00
GeeeekExplorer
bc0ad5a116
better
2025-06-17 23:33:38 +08:00
GeeeekExplorer
7e42fa6f63
fix
2025-06-15 13:28:29 +08:00
GeeeekExplorer
fc778a4da9
better
2025-06-15 10:36:45 +08:00
cheunglei
53b3ef2e32
support tensor parallel
2025-06-15 01:31:24 +08:00
GeeeekExplorer
4a8aa090a7
fix
2025-06-14 00:56:07 +08:00
GeeeekExplorer
f16adb729e
refactor
2025-06-12 09:41:12 +08:00
GeeeekExplorer
fee58d44e4
fix
2025-06-12 01:00:31 +08:00
GeeeekExplorer
386290d69e
refactor
2025-06-11 21:12:57 +08:00
GeeeekExplorer
b98e1ca305
fix
2025-06-10 21:25:54 +08:00
GeeeekExplorer
a5a4909e6a
init commit
2025-06-10 00:27:01 +08:00
