Commit Graph - vllm - Gitea: Git with a cup of tea

wassname/vllm

Fork 0

mirror of https://github.com/wassname/vllm.git synced 2026-09-21 13:40:06 +08:00

a046f86397 [Core/Bugfix] Add FP8 K/V Scale and dtype conversion for prefix/prefill Triton Kernel (#7208) jon-chuangandCody Yu 2024-08-12 15:47:41 -07:00
4ddc4743d7 [Core] Consolidate GB constant and enable float GB arguments (#7416) Cyrus Leung 2024-08-13 05:14:14 +08:00
6aa33cb2dd [Misc] Use scalar type to dispatch to different gptq_marlin kernels (#7323) Lucas Wilkinson 2024-08-12 14:40:13 -04:00
1137f343aa [ci] Cancel fastcheck when PR is ready (#7433) Kevin H. Luu 2024-08-12 10:59:14 -07:00
9b3e2edd30 [ci] Cancel fastcheck run when PR is marked ready (#7427) Kevin H. Luu 2024-08-12 10:56:52 -07:00
65950e8f58 [ci] Entrypoints run upon changes in vllm/ (#7423) Kevin H. Luu 2024-08-12 10:18:03 -07:00
cfba4def5d [Bugfix] Fix logit soft cap in flash-attn backend (#7425) Woosuk Kwon 2024-08-12 09:58:28 -07:00
d2bc4510a4 [CI/Build] bump Dockerfile.neuron image base, use public ECR (#6832) Daniele 2024-08-12 18:53:35 +02:00
24154f8618 [Frontend] Disallow passing model as both argument and option (#7347) Cyrus Leung 2024-08-12 20:58:34 +08:00
e6e42e4b17 [Core][VLM] Support image embeddings as input (#6613) Roger Wang 2024-08-12 01:16:06 -07:00
ec2affa8ae [Kernel] Flashinfer correctness fix for v0.1.3 (#7319) Lily Liu 2024-08-12 00:59:17 -07:00
86ab567bae [CI/Build] Minor refactoring for vLLM assets (#7407) Roger Wang 2024-08-11 19:41:52 -07:00
f020a6297e [Docs] Update readme (#7316) Simon Mo 2024-08-11 17:13:37 -07:00
6c8e595710 [misc] add commit id in collect env (#7405) youkaichao 2024-08-11 15:40:48 -07:00
02b1988b9f [Doc] building vLLM with VLLM_TARGET_DEVICE=empty (#7403) tomeras91 2024-08-12 00:38:17 +03:00
386087970a [CI/Build] build on empty device for better dev experience (#4773) tomeras91 2024-08-11 23:09:44 +03:00
c08e2b3086 [core] [2/N] refactor worker_base input preparation for multi-step (#7387) William Lin 2024-08-11 08:50:08 -07:00
4fb7b52a2c Updating LM Format Enforcer version to v0.10.6 (#7189) Noam Gat 2024-08-11 15:11:50 +03:00
90bab18f24 [TPU] Use mark_dynamic to reduce compilation time (#7340) Woosuk Kwon 2024-08-10 18:12:22 -07:00
4c5d8e8ea9 [Bugfix] Fix phi3v batch inference when images have different aspect ratio (#7392) Isotr0py 2024-08-11 00:19:33 +08:00
baa240252e [Core] Fix edge case in chunked prefill + block manager v2 (#7380) Cade Daniel 2024-08-09 16:48:49 -07:00
999ef0b917 [Misc] Add numpy implementation of compute_slot_mapping (#7377) Antoni Baum 2024-08-09 15:52:29 -07:00
5c6c54d67a [Bugfix] Fix PerTensorScaleParameter weight loading for fused models (#7376) Dipika Sikka 2024-08-09 17:23:46 -04:00
933790c209 [Core] Add span metrics for model_forward, scheduler and sampler time (#7089) Mahesh Keralapura 2024-08-09 13:55:13 -07:00
70d268a399 [Bugfix] Fix ITL recording in serving benchmark (#7372) Roger Wang 2024-08-09 10:00:00 -07:00
249b88228d [Frontend] Support embeddings in the run_batch API (#7132) Pooya DavoodiandSimon Mo 2024-08-09 09:48:21 -07:00
74af2bbd90 [Bugfix] Fix reinit procedure in ModelInputForGPUBuilder (#7360) Alexander Matveev 2024-08-09 12:35:49 -04:00
fc7b8d1eef [Performance] e2e overheads reduction: Small followup diff (#7364) Alexander Matveev 2024-08-09 11:49:36 -04:00
67abdbb42f [VLM][Doc] Add stop_token_ids to InternVL example (#7354) Isotr0py 2024-08-09 22:51:04 +08:00
07ab160741 [Model][Jamba] Mamba cache single buffer (#6739) Mor ZusmanandMor Zusman 2024-08-09 17:07:06 +03:00
b4e9528f95 [Core] Streamline stream termination in AsyncLLMEngine (#7336) Nick Hill 2024-08-09 00:06:36 -07:00
57b7be0e1c [Speculative decoding] [Multi-Step] decouple should_modify_greedy_probs_inplace (#6971) William Lin 2024-08-08 22:42:45 -07:00
99b4cf5f23 [Bugfix] Fix speculative decoding with MLPSpeculator with padded vocabulary (#7218) Travis Johnson 2024-08-08 23:08:46 -06:00
e02ac55617 [Performance] Optimize e2e overheads: Reduce python allocations (#7162) Alexander Matveev 2024-08-09 00:34:28 -04:00
73388c07a4 [TPU] Fix dockerfile.tpu (#7331) Woosuk Kwon 2024-08-08 20:24:58 -07:00
7eb4a51c5f [Core] Support serving encoder/decoder models (#7258) Cyrus Leung 2024-08-09 10:39:41 +08:00
0fa14907da [TPU] Add Load-time W8A16 quantization for TPU Backend (#7005) Siyuan Liu 2024-08-08 18:35:49 -07:00
5923532e15 Add Skywork AI as Sponsor (#7314) Simon Mo 2024-08-08 13:59:57 -07:00
a049b107e2 [Misc] Temporarily resolve the error of BitAndBytes (#7308) Jee Jee Li 2024-08-09 04:42:58 +08:00
8334c39f37 [Bugfix] Fix new Llama3.1 GGUF model loading (#7269) Isotr0py 2024-08-09 04:42:44 +08:00
e904576743 [CI/Build] Dockerfile.cpu improvements (#7298) Daniele 2024-08-08 21:24:52 +02:00
e14fb22e59 [Doc] Put collect_env issue output in a <detail> block (#7310) Michael Goin 2024-08-08 14:22:49 -04:00
782e53ab59 [Bugfix][fast] Fix the get_num_blocks_touched logic (#6849) Zach Zheng 2024-08-08 10:43:30 -07:00
21b9c49aa3 [Frontend] Kill the server on engine death (#6594) Joe Runde 2024-08-08 10:47:48 -06:00
5fb4a3f678 [Bugfix][Kernel] Increased atol to fix failing tests (#7305) Luka Govedič 2024-08-08 12:16:13 -04:00
757ac70a64 [Model] Rename MiniCPMVQwen2 to MiniCPMV2.6 (#7273) Jee Jee Li 2024-08-08 22:02:41 +08:00
6dffa4b0a6 [Bugfix] Fix LoRA with PP (#7292) Murali Andoorveedu 2024-08-08 00:02:27 -07:00
48abee9e54 [Frontend] remove max_num_batched_tokens limit for lora (#7288) Cherilyn Buren 2024-08-08 14:17:29 +08:00
746709642c [Misc] Fix typos in scheduler.py (#7285) Rui Qiao 2024-08-07 17:06:01 -07:00
e53dfd3eaf [Kernel] Fix Flashinfer Correctness (#7284) Lily Liu 2024-08-07 16:26:52 -07:00
6d94420246 [Doc] Update supported_hardware.rst (#7276) Michael Goin 2024-08-07 17:21:50 -04:00
fc1493a01e [FrontEnd] Make merge_async_iterators is_cancelled arg optional (#7282) Nick Hill 2024-08-07 13:35:14 -07:00
311f743831 [Bugfix] Fix gptq failure on T4s (#7264) Lucas Wilkinson 2024-08-07 16:05:37 -04:00
469b3bc538 [ci] Make building wheels per commit optional (#7278) Kevin H. Luu 2024-08-07 11:34:25 -07:00
5223199e03 [Bugfix][FP8] Fix dynamic FP8 Marlin quantization (#7219) Michael Goin 2024-08-07 14:23:12 -04:00
fde47d3bc2 [BugFix] Fix frontend multiprocessing hang (#7217) Maximilien de BayserandRobert Shaw 2024-08-07 15:09:36 -03:00
0e12cd67a8 [Doc] add online speculative decoding example (#7243) Stas Bekman 2024-08-07 09:58:02 -07:00
80cbe10c59 [OpenVINO] migrate to latest dependencies versions (#7251) Ilya Lavrenov 2024-08-07 20:49:10 +04:00
b764547616 [Bugfix] Fix input processor for InternVL2 model (#7164) Isotr0pyandCyrus Leung 2024-08-08 00:32:07 +08:00
ab0f5e2823 Fixes typo in function name (#7275) Rafael Vasquez 2024-08-07 12:29:27 -04:00
564985729a [ BugFix ] Move zmq frontend to IPC instead of TCP (#7222) Robert Shaw 2024-08-07 12:24:56 -04:00
0f7052bc7e [Misc] Refactor linear layer weight loading; introduce BasevLLMParameter and weight_loader_v2 (#5874) Dipika Sikka 2024-08-07 12:17:58 -04:00
639159b2a6 [distributed][misc] add specialized method for cuda platform (#7249) youkaichao 2024-08-07 08:54:52 -07:00
66d617e343 [Frontend] Gracefully handle missing chat template and fix CI failure (#7238) Cyrus LeungandRoger Wang 2024-08-07 17:12:05 +08:00
7b261092de [BUGFIX]: top_k is expected to be an integer. (#7227) Atilla Akkuş 2024-08-07 10:32:16 +03:00
2385c8f374 [Doc] Mock new dependencies for documentation (#7245) Roger Wang 2024-08-06 23:43:03 -07:00
9a3f49ae07 [BugFix] Overhaul async request cancellation (#7111) Nick Hill 2024-08-06 22:21:41 -07:00
f9a5600649 [Bugfix] Fix GPTQ and GPTQ Marlin CPU Offloading (#7225) Michael Goin 2024-08-06 21:34:26 -04:00
fd95e026e0 [Core] Subclass ModelRunner to support cross-attention & encoder sequences (towards eventual encoder/decoder model support) (#4942)
afeldman-nm Andrew Feldman Nick Hill
2024-08-06 16:51:47 -04:00
660470e5a3 [Core] Optimize evictor-v2 performance (#7193) xiaobochen123 2024-08-07 03:34:25 +08:00
8d59dbb000 [Kernel] Add per-tensor and per-token AZP epilogues (#5941) Luka GovedičandTyler Michael Smith 2024-08-06 14:17:08 -04:00
5c60c8c423 [SpecDecode] [Minor] Fix spec decode sampler tests (#7183) Lily Liu 2024-08-06 10:40:32 -07:00
00afc78590 [Bugfix] add gguf dependency (#7198) Katarzyna Papisandkatarzyna.papis 2024-08-06 19:08:35 +02:00
541c1852d3 [ BugFix ] Fix ZMQ when VLLM_PORT is set (#7205) Robert Shaw 2024-08-06 12:26:26 -04:00
a3bbbfa1d8 [BugFix] Fix DeepSeek remote code (#7178) Dipika Sikka 2024-08-06 11:16:53 -04:00
1f26efbb3a [Model] Support SigLIP encoder and alternative decoders for LLaVA models (#7153) Cyrus LeungandRoger Wang 2024-08-06 16:55:31 +08:00
9118217f58 [LoRA] Relax LoRA condition (#7146) Jee Jee Li 2024-08-06 09:57:25 +08:00
e3c664bfcb [Build] Add initial conditional testing spec (#6841) Simon Mo 2024-08-05 17:39:22 -07:00
360bd67cf0 [Core] Support loading GGUF model (#5191) Isotr0pyandMichael Goin 2024-08-06 07:54:23 +08:00
ef527be06c [MISC] Use non-blocking transfer in prepare_input (#7172) Cody Yu 2024-08-05 16:41:27 -07:00
89b8db6bb2 [Bugfix] Specify device when loading LoRA and embedding tensors (#7129) Jacob ScheinandJacob Schein 2024-08-05 16:35:47 -07:00
789937af2e [Doc] [SpecDecode] Update MLPSpeculator documentation (#7100) Thomas Parnell 2024-08-06 01:29:43 +02:00
dfb1a15dcb [ci][frontend] deduplicate tests (#7101) youkaichao 2024-08-05 15:59:22 -07:00
4db5176d97 bump version to v0.5.4 (#7139) Simon Mo 2024-08-05 14:39:48 -07:00
4cf1dc39be [Bugfix][CI/Build] Fix CUTLASS FetchContent (#7171) Tyler Michael Smith 2024-08-05 17:22:57 -04:00
6e4852ce28 [CI/Build] Suppress divide-by-zero and missing return statement warnings (#7001) Tyler Michael Smith 2024-08-05 16:00:01 -04:00
8571ac4672 [Kernel] Update CUTLASS to 3.5.1 (#7085) Tyler Michael Smith 2024-08-05 15:13:43 -04:00
997cf78308 [Misc] Fix typo in GroupCoordinator.recv() (#7167) Rui Qiao 2024-08-05 11:10:16 -07:00
57f560aa23 [BugFix] Use args.trust_remote_code (#7121) Aditya Paliwal 2024-08-05 09:26:14 -07:00
003f8ee128 [BugFix] Use IP4 localhost form for zmq bind (#7163) Nick Hill 2024-08-05 08:41:03 -07:00
e9630458c7 [SpecDecode] Support FlashInfer in DraftModelRunner (#6926) Bongwon Jang 2024-08-06 00:05:05 +09:00
82a1b1a82b [Speculative decoding] Add periodic log with time spent in proposal/scoring/verification (#6963) Cade Daniel 2024-08-05 01:46:44 -07:00
c0d8f1636c [Model] SiglipVisionModel ported from transformers (#6942) Jungho Christopher ChoandRoger Wang 2024-08-05 15:22:12 +09:00
cc08fc7225 [Frontend] Reapply "Factor out code for running uvicorn" (#7095) Cyrus Leung 2024-08-05 11:40:51 +08:00
7b86e7c9cd [Model] Add multi-image support for minicpmv (#7122)
Alphi hezhihui Cyrus Leung
2024-08-05 09:23:17 +08:00
f80ab3521c Clean up remaining Punica C information (#7027) Jee Jee Li 2024-08-05 06:37:08 +08:00
16a1cc9bb2 [misc][distributed] improve libcudart.so finding (#7127) youkaichao 2024-08-04 11:31:51 -07:00
b1c9aa3daa [Bugfix] [SpecDecode] Default speculative_draft_tensor_parallel_size to 1 when using MLPSpeculator (#7105) Thomas Parnell 2024-08-04 16:13:18 +02:00
179a6a36f2 [Model]Refactor MiniCPMV (#7020) Jee Jee LiandCyrus Leung 2024-08-04 16:12:41 +08:00
83c644fe7e [core][misc] simply output processing with shortcut code path (#7117) youkaichao 2024-08-04 00:22:19 -07:00

... 26 27 28 29 30 ...

Commit Graph Select branches Hide Pull Requests main Mono Color

Commit Graph

Select branches

Hide Pull Requests

main