Commit Graph - vllm - Gitea: Git with a cup of tea

wassname/vllm

Fork 0

mirror of https://github.com/wassname/vllm.git synced 2026-07-16 11:23:03 +08:00

f8ecb84c02 Speed up Punica compilation (#2632) Woosuk KwonandGitHub 2024-01-27 17:46:56 -08:00
5f036d2bcc [Minor] Fix warning on Ray dependencies (#2630) Woosuk KwonandGitHub 2024-01-27 15:43:40 -08:00
380170038e Implement custom all reduce kernels (#2192) Hanzhi ZhouandGitHub 2024-01-28 04:46:35 +08:00
220a47627b Use head_dim in config if exists (#2622) Xiang XuandGitHub 2024-01-27 10:30:49 -08:00
beb89f68b4 AWQ: Up to 2.66x higher throughput (#2566) CasperandGitHub 2024-01-27 08:53:17 +01:00
390b495ff3 Don't build punica kernels by default (#2605) Philipp MoritzandGitHub 2024-01-26 15:19:19 -08:00
3a0e1fc070 Support for Stable LM 2 (#2598)
dakotamahan-stability GitHub Zhuohan Li
2024-01-26 14:45:19 -06:00
6b7de1a030 [ROCm] add support to ROCm 6.0 and MI300 (#2274) Hongxia YangandGitHub 2024-01-26 15:41:10 -05:00
5265631d15 use a correct device when creating OptionalCUDAGuard (#2583) VladimirandGitHub 2024-01-26 08:48:17 +01:00
2832e7b9f9 fix names and license for Qwen2 (#2589) Junyang LinandGitHub 2024-01-25 14:37:51 +08:00
3a7dd7e367 Support Batch Completion in Server (#2529) Simon MoandGitHub 2024-01-24 17:11:07 -08:00
223c19224b Fix the syntax error in the doc of supported_models (#2584) LastWhisperandGitHub 2024-01-25 03:22:51 +08:00
f1f6cc10c7 Added include_stop_str_in_output and length_penalty parameters to OpenAI API (#2562) Federico GalatoloandGitHub 2024-01-24 19:21:56 +01:00
3209b49033 [Bugfix] fix crash if max_tokens=None (#2570) Nikola BorisovandGitHub 2024-01-23 22:38:55 -08:00
1e4277d2d1 lint: format all python file instead of just source code (#2567) Simon MoandGitHub 2024-01-23 15:53:06 -08:00
9b945daaf1 [Experimental] Add multi-LoRA support (#1804)
Antoni Baum GitHub Chen Shen Shreyas Krishnaswamy Avnish Narayan
2024-01-24 00:26:37 +01:00
9c1352eb57 [Feature] Simple API token authentication and pluggable middlewares (#1106) Erfan Al-HossamiandGitHub 2024-01-23 18:13:00 -05:00
7a0b011dd5 Add a 1-line docstring to explain why calling context_attention_fwd twice in test_prefix_prefill.py (#2553) Jason ZhuandGitHub 2024-01-22 14:47:25 -08:00
63e835cbcc Fix progress bar and allow HTTPS in benchmark_serving.py (#2552) Harry MellorandGitHub 2024-01-22 22:40:31 +00:00
94b5edeb53 Add qwen2 (#2495) Junyang LinandGitHub 2024-01-23 06:34:21 +08:00
ab7e6006d6 Fix https://github.com/vllm-project/vllm/issues/2540 (#2545) Philipp MoritzandGitHub 2024-01-22 10:02:38 -08:00
18bfcdd05c [Speculative decoding 2/9] Multi-step worker for draft model (#2424) Cade DanielandGitHub 2024-01-21 16:31:47 -08:00
71d63ed72e migrate pydantic from v1 to v2 (#2531) Jannis SchönleberandGitHub 2024-01-22 01:05:56 +01:00
d75c40734a [Fix] Keep scheduler.running as deque (#2523) Nick HillandGitHub 2024-01-20 22:36:09 -08:00
5b23c3f26f Add group as an argument in broadcast ops (#2522) Junda ChenandGitHub 2024-01-20 16:00:26 -08:00
00efdc84ba Add benchmark serving to CI (#2505) Simon MoandGitHub 2024-01-19 20:20:19 -08:00
91a61da9b1 [Bugfix] fix load local safetensors model (#2512) RoyandGitHub 2024-01-20 08:26:16 +08:00
ef9b636e2d Simplify broadcast logic for control messages (#2501) Zhuohan LiandGitHub 2024-01-19 11:23:30 -08:00
2709c0009a Support OpenAI API server in benchmark_serving.py (#2172) Harry MellorandGitHub 2024-01-19 04:34:08 +00:00
dd7e8f5f64 refactor complemention api for readability (#2499) Simon MoandGitHub 2024-01-18 16:45:14 -08:00
d2a68364c4 [BugFix] Fix abort_seq_group (#2463) ljssandGitHub 2024-01-19 07:10:42 +08:00
7e1081139d Don't download both safetensor and bin files. (#2480) Nikola BorisovandGitHub 2024-01-18 11:05:53 -08:00
18473cf498 [Neuron] Add an option to build with neuron (#2065) Liangfu ChenandGitHub 2024-01-18 10:58:50 -08:00
4df417d059 fix: fix some args desc (#2487) zspoandGitHub 2024-01-19 01:41:44 +08:00
5d80a9178b Minor fix in prefill cache example (#2494) Jason ZhuandGitHub 2024-01-18 09:40:34 -08:00
8a25d3a71a fix stablelm.py tensor-parallel-size bug (#2482) YingchaoXandGitHub 2024-01-19 01:39:46 +08:00
d10f8e1d43 [Experimental] Prefix Caching Support (#1669)
shiyi.c_98 GitHub DouHappy Zhuohan Li
2024-01-17 16:32:10 -08:00
14cc317ba4 OpenAI Server refactoring (#2360) FlorianJoncourandGitHub 2024-01-17 05:33:14 +00:00
e1957c6ebd Add StableLM3B model (#2372) Hyunsung LeeandGitHub 2024-01-17 13:32:40 +09:00
8cd5a992bf ci: retry on build failure as well (#2457) Simon MoandGitHub 2024-01-16 12:51:04 -08:00
947f0b23cc CI: make sure benchmark script exit on error (#2449) Simon MoandGitHub 2024-01-16 09:50:13 -08:00
f780504d12 fix weigit loading for GQA with TP (#2379) Chenhui ZhangandGitHub 2024-01-16 07:43:59 +08:00
bfc072addf Allow buildkite to retry build on agent lost (#2446) Simon MoandGitHub 2024-01-15 15:43:15 -08:00
2a18da257c Announce the second vLLM meetup (#2444) Woosuk KwonandGitHub 2024-01-15 14:11:59 -08:00
6e01e8c1c8 [CI] Add Buildkite (#2355) Simon MoandGitHub 2024-01-14 12:37:58 -08:00
9f659bf07f [Minor] Optimize cuda graph memory usage (#2437) RoyandGitHub 2024-01-15 01:40:51 +08:00
35c4bc20d9 [Minor] Fix err msg (#2431) Woosuk KwonandGitHub 2024-01-12 14:02:52 -08:00
218dc2ccda Aligning top_p and top_k Sampling (#1885) 陈序andGitHub 2024-01-13 05:51:03 +08:00
827cbcd37c Update quickstart.rst (#2369) SimonandGitHub 2024-01-12 14:56:18 -06:00
cb7a1c1cbf Suggest using dtype=half when OOM. BenandGitHub 2024-01-13 04:33:29 +08:00
7878958c0d Address Phi modeling update 2 (#2428) Gary HuiandGitHub 2024-01-13 04:16:49 +08:00
ce036244c9 Allow setting fastapi root_path argument (#2341) Chirag JainandGitHub 2024-01-13 00:29:59 +05:30
48cf1e413c fix: deque mutated during iteration in abort_seq_group (#2371) 陈序andGitHub 2024-01-13 00:44:18 +08:00
97460585d9 Add gradio chatbot for openai webserver (#2307) arkohutandGitHub 2024-01-12 11:45:56 +08:00
f745847ef7 [Minor] Fix the format in quick start guide related to Model Scope (#2425) Zhuohan LiandGitHub 2024-01-11 19:44:01 -08:00
6549aef245 [DOC] Add additional comments for LLMEngine and AsyncLLMEngine (#1011) JiaxiangandGitHub 2024-01-12 11:26:49 +08:00
50376faa7b Rename phi_1_5 -> phi (#2385) Woosuk KwonandGitHub 2024-01-11 16:23:43 -08:00
4b61c6b669 get_ip(): Fix ipv4 ipv6 dualstack (#2408) Yunfeng BaiandGitHub 2024-01-10 11:39:58 -08:00
79d64c4954 [Speculative decoding 1/9] Optimized rejection sampler (#2336) Cade DanielandGitHub 2024-01-09 15:38:41 -08:00
74cd5abdd1 Add baichuan chat template jinjia file (#2390) KKYandGitHub 2024-01-09 11:13:02 -06:00
28c3f12104 [Minor] Remove unused code in attention (#2384) Woosuk KwonandGitHub 2024-01-08 13:13:08 -08:00
c884819135 Fix eager mode performance (#2377) Woosuk KwonandGitHub 2024-01-08 10:11:06 -08:00
05921a9a7a Changed scheduler to use deques instead of lists (#2290)
Nadav Shmayovits GitHub Woosuk Kwon
2024-01-07 19:48:07 +02:00
d0215a58e7 Ensure metrics are logged regardless of requests (#2347) Iskren Ivov ChernevandGitHub 2024-01-05 15:24:42 +02:00
937e7b7d7c Build docker image with shared objects from "build" step (#2237) Alexandre PayotandGitHub 2024-01-04 18:35:18 +01:00
aee8ef661a Miner fix of type hint (#2340) ljssandGitHub 2024-01-04 13:27:56 +08:00
2e0b6e7757 Bump up to v0.2.7 (#2337) Woosuk KwonandGitHub 2024-01-03 17:35:56 -08:00
941767127c Revert the changes in test_cache (#2335) Woosuk KwonandGitHub 2024-01-03 17:32:05 -08:00
74d8d77626 Remove unused const TIMEOUT_TO_PREVENT_DEADLOCK (#2321) Ronen SchafferandGitHub 2024-01-04 01:49:07 +02:00
fd4ea8ef5c Use NCCL instead of ray for control-plane communication to remove serialization overhead (#2221) Zhuohan LiandGitHub 2024-01-04 03:30:22 +08:00
1066cbd152 Remove deprecated parameter: concurrency_count (#2315) Ronen SchafferandGitHub 2024-01-03 19:56:21 +02:00
6ef00b03a2 Enable CUDA graph for GPTQ & SqueezeLLM (#2318) Woosuk KwonandGitHub 2024-01-03 09:52:29 -08:00
9140561059 [Minor] Fix typo and remove unused code (#2305) RoyandGitHub 2024-01-03 11:23:15 +08:00
77af974b40 [FIX] Support non-zero CUDA devices in custom kernels (#1959) Jee LiandGitHub 2024-01-03 11:09:59 +08:00
4934d49274 Support GPT-NeoX Models without attention biases (#2301) Jong-hun ShinandGitHub 2023-12-31 01:42:04 +09:00
358c328d69 [BUGFIX] Fix communication test (#2285) Zhuohan LiandGitHub 2023-12-28 06:18:11 +08:00
4aaafdd289 [BUGFIX] Fix the path of test prompts (#2273) Zhuohan LiandGitHub 2023-12-27 02:37:21 +08:00
66b108d142 [BUGFIX] Fix API server test (#2270) Zhuohan LiandGitHub 2023-12-27 02:37:06 +08:00
e0ff920001 [BUGFIX] Do not return ignored sentences twice in async llm engine (#2258) Zhuohan LiandGitHub 2023-12-26 13:41:09 +08:00
face83c7ec [Docs] Add "About" Heading to README.md (#2260) blueceilingandGitHub 2023-12-25 17:37:07 -07:00
1db83e31a2 [Docs] Update installation instructions to include CUDA 11.8 xFormers (#2246) Shivam ThakkarandGitHub 2023-12-23 12:50:02 +05:30
a1b9cb2a34 [BugFix] Fix recovery logic for sequence group (#2186) Woosuk KwonandGitHub 2023-12-20 21:52:37 -08:00
3a4fd5ca59 Disable Ray usage stats collection (#2206) Woosuk KwonandGitHub 2023-12-20 21:52:08 -08:00
c17daa9f89 [Docs] Fix broken links (#2222) Ronen SchafferandGitHub 2023-12-20 22:43:42 +02:00
bd29cf3d3a Remove Sampler copy stream (#2209) Antoni BaumandGitHub 2023-12-20 00:04:33 -08:00
31bff69151 Make _prepare_sample non-blocking and use pinned memory for input buffers (#2207) Hanzhi ZhouandGitHub 2023-12-19 16:52:46 -08:00
ba4f826738 [BugFix] Fix weight loading for Mixtral with TP (#2208) Woosuk KwonandGitHub 2023-12-19 16:16:11 -08:00
de60a3fb93 Added DeciLM-7b and DeciLM-7b-instruct (#2062) avideciandGitHub 2023-12-19 12:29:33 +02:00
21d5daa4ac Add warning on CUDA graph memory usage (#2182) Woosuk KwonandGitHub 2023-12-18 18:16:17 -08:00
290e015c6c Update Help Text for --gpu-memory-utilization Argument (#2183) Suhong MoonandGitHub 2023-12-18 14:33:24 -05:00
1b7c791d60 [ROCm] Fixes for GPTQ on ROCm (#2180) kliuaeandGitHub 2023-12-19 02:41:04 +08:00
bbe4466fd9 [Minor] Fix typo (#2166)
JohnSaxon GitHub John-Saxon
2023-12-18 15:28:49 +08:00
08133c4d1a Add SSL arguments to API servers (#2109) Harry MellorandGitHub 2023-12-18 02:56:23 +00:00
76a7983b23 [BugFix] Fix RoPE kernel on long sequences(#2164) Woosuk KwonandGitHub 2023-12-17 17:09:10 -08:00
8041b7305e [BugFix] Raise error when max_model_len is larger than KV cache (#2163) Woosuk KwonandGitHub 2023-12-17 17:08:23 -08:00
3ec8c25cd0 [Docs] Update documentation for gpu-memory-utilization option (#2162) Suhong MoonandGitHub 2023-12-17 13:51:57 -05:00
671af2b1c0 Bump up to v0.2.6 (#2157) Woosuk KwonandGitHub 2023-12-17 10:34:56 -08:00
6f41f0e377 Disable CUDA graph for SqueezeLLM (#2161) Woosuk KwonandGitHub 2023-12-17 10:24:25 -08:00
2c9b638065 [Minor] Fix a typo in .pt weight support (#2160) Woosuk KwonandGitHub 2023-12-17 10:12:44 -08:00
a7347d9a6d Make sampler less blocking (#1889) Antoni BaumandGitHub 2023-12-17 07:03:49 -08:00

... 42 43 44 45 46 ...

Commit Graph Select branches Hide Pull Requests main Mono Color

Commit Graph

Select branches

Hide Pull Requests

main