wassname/vllm
1
0
Fork 0
mirror of https://github.com/wassname/vllm.git synced 2026-08-15 12:54:49 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,562 Commits 1 Branch 0 Tags
bc55d13070a9f3f2b8524901817d406cb8a37a3b
Commit Graph
1356 Commits
Author SHA1 Message Date
Isotr0py bc55d13070 [VLM] Implement merged multimodal processor for Mllama (#11427) 2025-02-12 20:26:21 -08:00
Kaixi Hou 4fc5c23bb6 [NVIDIA] Support nvfp4 quantization (#12784) 2025-02-12 19:51:51 -08:00
Michael Goin 14b7899d10 [CI] Fix failing FP8 cpu offload test (#13170) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-12 19:16:06 +00:00
Qubitium-ModelCloud 36a08630e8 [CORE] [QUANT] Support for GPTQModel's dynamic quantization per module override/control (#7086) 2025-02-12 09:19:43 -08:00
Jee Jee Li 82cabf53a3 [Misc] Delete unused LoRA modules (#13151) 2025-02-12 08:58:24 -08:00
Rafael Vasquez 314cfade02 [Frontend] Generate valid tool call IDs when using tokenizer-mode=mistral (#12332) 2025-02-12 08:29:56 -08:00
Lingfan Yu e92694b6fe [Neuron][Kernel] Support Longer Sequences in NKI-based Flash PagedAttention and Improve Efficiency (#12921) 
Signed-off-by: Lingfan Yu <lingfany@amazon.com>
 2025-02-11 21:12:37 -08:00
Christian Pinto 974dfd4971 [Model] IBM/NASA Prithvi Geospatial model (#12830) 2025-02-11 20:34:30 -08:00
Keyun Tong 3ee696a63d [RFC][vllm-API] Support tokenizer registry for customized tokenizer in vLLM (#12518) 
Signed-off-by: Keyun Tong <tongkeyun@gmail.com>
 2025-02-12 12:25:58 +08:00
ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 6c4dbe23eb [BugFix] Pop instead of del CUDA_VISIBLE_DEVICES (#12962) 
Signed-off-by: Hollow Man <hollowman@opensuse.org>
 2025-02-12 00:21:50 +08:00
Mark McLoughlin 75e6e14516 [V1][Metrics] Add several request timing histograms (#12644) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-02-11 10:14:00 -05:00
மனோஜ்குமார் பழனிச்சாமி 110f59a33e [Bugfix] fix flaky test (#13089) 
Signed-off-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
 2025-02-11 14:41:20 +00:00
Cody Yu 41c5dd45b9 [V1][Metrics] Add GPU prefix cache hit rate % gauge (#12592) 2025-02-11 08:27:25 +00:00
Ce Gao fc6485d277 [Bugfix]: Reasoning output bug according to the chat template change (#13025) 
Signed-off-by: Ce Gao <cegao@tensorchord.ai>
 2025-02-11 15:49:03 +08:00
Varun Sundar RabindranathandVarun Sundar Rabindranath 78a141d768 [Misc] LoRA - Refactor Punica ops tests (#12970) 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
 2025-02-11 07:26:03 +00:00
Florian Greinacher cb080f32e3 [Bugfix] Support missing tool parameters in mistral tokenizer (#12884) 
Signed-off-by: Florian Greinacher <florian.greinacher@siemens.com>
 2025-02-11 03:33:33 +00:00
Farzad Abdolhosseini 08b2d845d6 [Model] Ultravox Model: Support v0.5 Release (#12912) 
Signed-off-by: Farzad Abdolhosseini <farzad@fixie.ai>
 2025-02-10 22:02:48 +00:00
மனோஜ்குமார் பழனிச்சாமி 2ae889052c Fix seed parameter behavior in vLLM (#13007) 
Signed-off-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
 2025-02-10 23:26:50 +08:00
Cyrus Leung 51f0b5f7f6 [Bugfix] Clean up and fix multi-modal processors (#13012) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-02-10 10:45:21 +00:00
youkaichao b2496bb07f [core] fix sleep mode and pytorch checkpoint compatibility (#13001) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-10 13:03:43 +08:00
youkaichao 59fff4a01a [core] improve error handling when wake up from sleep mode (#12981) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-10 09:38:57 +08:00
youkaichao cf797aa856 [core] port pynvml into vllm codebase (#12963) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-09 15:00:00 +08:00
Jee Jee Li 86222a3dab [VLM] Merged multi-modal processor for GLM4V (#12449) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-02-08 20:32:16 +00:00
youkaichao 91dd8f7aa6 [bugfix] respect distributed_executor_backend in world_size=1 (#12934) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-08 16:17:08 +08:00
Woosuk Kwon 3243158336 [V1] Move KV block hashes from Request to KVCacheManager (#12922) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-07 19:14:10 -08:00
TJian eaa92d4437 [ROCm] [Feature] [Doc] [Dockerfile] [BugFix] Support Per-Token-Activation Per-Channel-Weight FP8 Quantization Inferencing (#12501) 2025-02-07 08:13:43 -08:00
afeldman-nmrshaw@neuralmagic.com <rshaw@neuralmagic.com>Nick Hill
 		0630d4537a [V1] Logprobs and prompt logprobs support (#9880) 
This PR is adding support for sample logprobs & prompt logprobs to vLLM v1.

New behavior:

- During model execution, model runner computes sample logprobs (if user-provided logprobs setting is not None) and prompt logprobs (if user-provided prompt_logprobs setting is not None). For both sample and prompt logprobs, the engine core returns 3 vectors: token ids, token logprob values, token ranks. Ranks reflect tokens' 1-indexed positions in the vocabulary vector after sorting the vocabulary by log probability in descending order.
- In scheduler.update_from_output(), sample and prompt logprobs are incorporated into the EngineCoreOutput data structure which is transferred to the engine client. If multiprocessing is enabled, then sample and prompt logprobs will be (de)serialized when the EngineCoreOutput data structure is (de)serialized.
- During output processing, the LogprobsProcessor transforms the triplet of token ids, token logprobs values, and token ranks into the OpenAI-compatible List[Dict[token id,Logprob]] format (for sample and prompt logprobs respectively.)
- Each Logprob instance (whether sample- or prompt-) consists of a token's log-probability, rank, and detokenized string representation. Note that logprob detokenization is handled by the LogprobsProcessor not the detokenizer.

Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>


Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-02-07 07:26:20 -08:00
Cyrus Leung ce26b16268 [Misc] Remove unnecessary detokenization in multimodal processing (#12868) 2025-02-07 06:21:17 -08:00
Maximilien de Bayser 6e1fc61f0f Prevent unecessary requests to huggingface hub (#12837) 2025-02-06 21:37:41 -08:00
Yu Chin Fabian LimandTyler Michael Smith aff404571b Add Bamba Model (#10909) 
Signed-off-by: Yu Chin Fabian Lim <flim@sg.ibm.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-02-06 15:22:42 -08:00
Varun Sundar RabindranathandVarun Sundar Rabindranath 467a96a541 [V1] LoRA Support (#10957) 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
 2025-02-06 09:32:51 -08:00
youkaichao 09b95e36ab [torch.compile] PyTorch 2.6 and nightly compatibility (#12393) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-07 01:09:07 +08:00
Isotr0py 85ac82d228 [Kernel] Make rotary_embedding ops more flexible with input shape (#12777) 2025-02-06 08:46:13 -08:00
Dipika Sikka 7ca9934fe7 [Misc] Update w2 scale loading for GPTQMarlinMoE (#12757) 2025-02-06 01:02:14 -08:00
Lu Fang 56534cd577 [Bugfix] Fix the test_ultravox.py's license (#12806) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-02-06 13:25:54 +08:00
Sumit Vij d88506dda4 [Model] LoRA Support for Ultravox model (#11253) 2025-02-05 19:54:13 -08:00
Cyrus Leung 75404d041b [VLM] Update compatibility with transformers 4.49 2025-02-05 19:09:45 -08:00
Roger Wang bf3b79efb8 [VLM] Qwen2.5-VL 2025-02-05 13:31:38 -08:00
Rahul Tuli 3b2005e1db Add: Support for Sparse24Bitmask Compressed Models 2025-02-05 13:30:43 -08:00
Lucas Wilkinsonandsimon-mo 75e94309e8 [Perf] Mem align KV caches for CUDA devices (MLA perf improvement) (#12676) 
Signed-off-by: simon-mo <xmo@berkeley.edu>
Signed-off-by: Lucas Wilkinson <lcwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-02-04 18:22:24 -08:00
Mark McLoughlin 233df6f5c4 [V1][Metrics] Add request_success_total counter, labelled with finish reason (#12579) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-02-04 19:46:54 -05:00
Cyrus Leung 18016a5e62 [Bugfix] Fix CI failures for InternVL and Mantis models (#12728) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-02-04 23:54:23 +08:00
Isotr0pyandCyrus Leung 815079de8e [VLM] merged multimodal processor and V1 support for idefics3 (#12660) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-02-04 20:00:51 +08:00
Woosuk Kwon 18a88fcccc [V1] Remove scheduling constraint on partial requests (#12674) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-04 02:43:58 -08:00
Cyrus LeungandIsotr0py d1ca7df84d [VLM] Merged multi-modal processor for InternVL-based models (#12553) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
 2025-02-04 16:44:52 +08:00
Hongxia YangandMatthew Wong c36ac98d01 [AMD][ROCm] Enable DeepSeek model on ROCm (#12662) 
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
 2025-02-04 08:24:11 +00:00
Thomas Parnell bb392af434 [Doc] Replace ibm-fms with ibm-ai-platform (#12709) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2025-02-04 07:05:04 +00:00
Russell Bryant 73b35cca7f [Core] Improve hash collision avoidance in prefix caching (#12621) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-03 16:28:20 -08:00
Cody Yu 5095e96606 [V1] Revert uncache_blocks and support recaching full blocks (#12415) 
Signed-off-by: Cody Yu <hao.yu.cody@gmail.com>
 2025-02-03 15:04:53 -08:00
Cody Yu cf58b9c4ca [MISC] Remove model input dumping when exception (#12582) 
Signed-off-by: Cody Yu <hao.yu.cody@gmail.com>
 2025-02-03 13:34:16 -08:00