Commit Graph

Select branches

Hide Pull Requests

main

Mono Color

• 3123f15138 Fixes the incorrect argument in the prefix-prefill test cases (#3246) Tao HeandGitHub 2024-03-16 11:58:10 +08:00
• 413366e9a2 [Misc] PR templates (#3413) 3 people 2024-03-15 18:25:51 -07:00
• 10585e035e Removed Extraneous Print Message From OAI Server (#3440) Robert ShawandGitHub 2024-03-15 19:35:36 -05:00
• fb96c1e98c Asynchronous tokenization (#2879) Antoni BaumandGitHub 2024-03-15 16:37:01 -07:00
• 8fa7357f2d fix document error for value and v_vec illustration (#3421) laneeeeandGitHub 2024-03-16 07:06:09 +08:00
• a7af4538ca Fix issue templates (#3436) Harry MellorandGitHub 2024-03-15 21:26:00 +00:00
• 604f235937 [Misc] add error message in non linux platform (#3438) youkaichaoandGitHub 2024-03-15 14:21:37 -07:00
• 14b8ae02e7 Fixes the misuse/mixuse of time.time()/time.monotonic() (#3220) 3 people 2024-03-16 02:25:43 +08:00
• 03d37f2441 [Fix] Add args for mTLS support (#3430) 3 people 2024-03-15 09:56:13 -07:00
• a7c871680e Fix tie_word_embeddings for Qwen2. (#3344) Yang FanandGitHub 2024-03-16 00:36:53 +08:00
• 429284dc37 Fix dist.broadcast stall without group argument (#3408) Junda ChenandGitHub 2024-03-14 23:25:05 -07:00
• 253a98078a Add chat templates for ChatGLM (#3418) Dinghow YangandGitHub 2024-03-15 14:19:22 +08:00
• 21539e6856 Add chat templates for Falcon (#3420) Dinghow YangandGitHub 2024-03-15 14:19:02 +08:00
• b522c4476f [Misc] add HOST_IP env var (#3419) 3 people 2024-03-14 21:32:52 -07:00
• 78b6c4845a Dynamically configure shared memory size for moe_align_block_size_kernel (#3376) akhoroshevandGitHub 2024-03-15 04:18:07 +03:00
• b983ba35bd fix marlin config repr (#3414) Enrique ShockwaveandGitHub 2024-03-14 23:26:19 +00:00
• 54be8a0be2 Fix assertion failure in Qwen 1.5 with prefix caching enabled (#3373) 3 people 2024-03-15 04:56:57 +08:00
• dfc77408bd [issue templates] add some issue templates (#3412) youkaichaoandGitHub 2024-03-14 13:16:00 -07:00
• c17ca8ef18 Add args for mTLS support (#3410) 3 people 2024-03-14 13:11:45 -07:00
• 06ec486794 Install flash_attn in Docker image (#3396) Thomas ParnellandGitHub 2024-03-14 18:55:54 +01:00
• 8fe8386591 [Kernel] change benchmark script so that result can be directly used; tune moe kernel in A100/H100 with tp=2,4,8 (#3389) youkaichaoandGitHub 2024-03-14 01:11:48 -07:00
• a37415c31b allow user to chose which vllm's merics to display in grafana (#3393) Allen.DouandGitHub 2024-03-14 14:35:13 +08:00
• 81653d9688 [Hotfix] [Debug] test_openai_server.py::test_guided_regex_completion (#3383) Simon MoandGitHub 2024-03-13 17:02:21 -07:00
• eeab52a4ff [FIX] Simpler fix for async engine running on ray (#3371) Zhuohan LiandGitHub 2024-03-13 14:18:40 -07:00
• c33afd89f5 Fix lint (#3388) Antoni BaumandGitHub 2024-03-13 13:56:49 -07:00
• 7e9bd08f60 Add batched RoPE kernel (#3095) TerryandGitHub 2024-03-13 13:45:26 -07:00
• ae0ccb4017 Add missing kernel for CodeLlama-34B on A/H100 (no tensor parallelism) when using Multi-LoRA. (#3350) Or SharirandGitHub 2024-03-13 21:18:25 +02:00
• 739c350c19 [Minor Fix] Use cupy-cuda11x in CUDA 11.8 build (#3256) 陈序andGitHub 2024-03-14 00:43:24 +08:00
• ba8dc958a3 [Minor] Fix bias in if to remove ambiguity (#3259) Hui LiuandGitHub 2024-03-13 09:16:55 -07:00
• e221910e77 add hf_transfer to requirements.txt (#3031) Ronan McGovernandGitHub 2024-03-13 06:33:43 +00:00
• b167109ba1 [Fix] Fix quantization="gptq" when using Marlin (#3319) 3 people 2024-03-13 13:51:42 +08:00
• 602358f8a8 Add kernel for GeGLU with approximate GELU (#3337) Woosuk KwonandGitHub 2024-03-12 22:06:17 -07:00
• 49a3c8662b Fixes #1556 double free (#3347) Breno FariaandGitHub 2024-03-13 01:30:08 +01:00
• b0925b3878 docs: Add BentoML deployment doc (#3336) Sherlock XuandGitHub 2024-03-13 01:34:30 +08:00
• 654865e21d Support Mistral Model Inference with transformers-neuronx (#3153) DAIZHENWEIandGitHub 2024-03-11 13:19:51 -07:00
• c9415c19d3 [ROCm] Fix warp and lane calculation in blockReduceSum (#3321) kliuaeandGitHub 2024-03-12 04:14:07 +08:00
• 4c922709b6 Add distributed model executor abstraction (#3191) Zhuohan LiandGitHub 2024-03-11 11:03:45 -07:00
• 657061fdce [docs] Add LoRA support information for models (#3299) Philipp MoritzandGitHub 2024-03-11 00:54:51 -07:00
• 2f8844ba08 Re-enable the 80 char line width limit (#3305) Zhuohan LiandGitHub 2024-03-10 19:49:14 -07:00
• 4b59f00e91 [Fix] Fix best_of behavior when n=1 (#3298) Nick HillandGitHub 2024-03-10 19:17:46 -07:00
• 9e8744a545 [BugFix] Fix get tokenizer when using ray (#3301) RoyandGitHub 2024-03-11 10:17:16 +08:00
• e4a28e5316 [ROCM] Fix blockReduceSum to use correct warp counts for ROCm and CUDA (#3262) Douglas LehrandGitHub 2024-03-10 17:27:45 -05:00
• 0bba88df03 Enhance lora tests with more layer and rank variations (#3243) TerryandGitHub 2024-03-09 17:14:16 -08:00
• 8437bae6ef [Speculative decoding 3/9] Worker which speculates, scores, and applies rejection sampling (#3103) Cade DanielandGitHub 2024-03-08 23:32:46 -08:00
• f48c6791b7 [FIX] Fix prefix test error on main (#3286) Zhuohan LiandGitHub 2024-03-08 17:16:14 -08:00
• c2c5e0909a Move model filelocks from /tmp/ to ~/.cache/vllm/locks/ dir (#3241) Michael GoinandGitHub 2024-03-08 13:33:10 -08:00
• 1cb0cc2975 [FIX] Make flash_attn optional (#3269) Woosuk KwonandGitHub 2024-03-08 10:52:20 -08:00
• 99c3cfb83c [Docs] Fix Unmocked Imports (#3275) Roger WangandGitHub 2024-03-08 09:58:01 -08:00
• 1ece1ae829 [Minor Fix] Fix comments in benchmark_serving (#3252) TianYu GUOandGitHub 2024-03-08 14:22:59 +08:00
• c59e120c55 Feature add lora support for Qwen2 (#3177) whyiugandGitHub 2024-03-08 13:58:24 +08:00
• d2339d6840 Connect engine healthcheck to openai server (#3260) Nick HillandGitHub 2024-03-07 16:38:12 -08:00
• b35cc93420 Fix auto prefix bug (#3239) ElizaWszolaandGitHub 2024-03-08 01:37:28 +01:00
• 8cbba4622c Possible fix for conflict between Automated Prefix Caching (#2762) and multi-LoRA support (#1804) (#3263) jacobthebananaandGitHub 2024-03-07 18:03:22 -05:00
• 385da2dae2 Measure model memory usage (#3120) Michael GoinandGitHub 2024-03-07 11:42:42 -08:00
• 2daf23ab0c Separate attention backends (#3005) Woosuk KwonandGitHub 2024-03-07 01:45:50 -08:00
• cbf4c05b15 Update requirements-dev.txt to include package for benchmarking scripts. (#3181) 3 people 2024-03-07 03:39:28 -05:00
• d3c04b6a39 Add GPTQ support for Gemma (#3200) TechxGenusandGitHub 2024-03-07 08:19:14 +08:00
• 4cb3b924cd Add tqdm dynamic_ncols=True (#3242) Chujie ZhengandGitHub 2024-03-06 14:41:42 -08:00
• a33ce60c66 [Testing] Fix core tests (#3224) Cade DanielandGitHub 2024-03-06 01:04:23 -08:00
• 24aecf421a [Tests] Add block manager and scheduler tests (#3108) SangBin ChoandGitHub 2024-03-06 11:23:34 +09:00
• 2efce05dc3 [Fix] Avoid pickling entire LLMEngine for Ray workers (#3207) 3 people 2024-03-05 16:17:20 -08:00
• 8999ec3c16 Store eos_token_id in Sequence for easy access (#3166) Nick HillandGitHub 2024-03-05 15:35:43 -08:00
• 05af6da8d9 [ROCm] enable cupy in order to enable cudagraph mode for AMD GPUs (#3123) 3 people 2024-03-04 21:14:53 -05:00
• 9a4548bae7 Fix the openai benchmarking requests to work with latest OpenAI apis (#2992) 3 people 2024-03-04 18:51:56 -05:00
• ff578cae54 Add health check, make async Engine more robust (#3015) 3 people 2024-03-04 14:01:40 -08:00
• 22de45235c Push logprob generation to LLMEngine (#3065) 3 people 2024-03-04 11:54:06 -08:00
• 76e8a70476 [Minor fix] The domain dns.google may cause a socket.gaierror exception (#3176) 3 people 2024-03-05 03:17:12 +08:00
• 9cbc7e5f3b enable --gpu-memory-utilization in benchmark_throughput.py (#3175) 3 people 2024-03-05 02:37:58 +08:00
• 27a7b070db Add document for vllm paged attention kernel. (#2978) Jialun LyuandGitHub 2024-03-04 09:23:34 -08:00
• 901cf4c52b [Minor Fix] Remove unused code in benchmark_prefix_caching.py (#3171) TianYu GUOandGitHub 2024-03-04 14:48:27 +08:00
• d0fae88114 [DOC] add setup document to support neuron backend (#2777) Liangfu ChenandGitHub 2024-03-03 17:03:51 -08:00
• 17c3103c56 Make it easy to profile workers with nsight (#3162) 3 people 2024-03-03 16:19:13 -08:00
• 996d095c54 [FIX] Fix styles in automatic prefix caching & add a automatic prefix caching benchmark (#3158) Zhuohan LiandGitHub 2024-03-03 14:37:18 -08:00
• d65fac2738 Add vLLM version info to logs and openai API server (#3161) Jason CoxandGitHub 2024-03-03 00:00:29 -05:00
• ce4f5a29fb Add Automatic Prefix Caching (#2762) 4 people 2024-03-02 03:50:01 -05:00
• baee28c46c Reorder kv dtype check to avoid nvcc not found error on AMD platform (#3104) cloudhanandGitHub 2024-03-02 14:34:48 +08:00
• 29e70e3e88 allow user chose log level by --log-level instead of fixed 'info'. (#3109) 4 people 2024-03-02 07:28:41 +08:00
• 82091b864a Bump up to v0.3.3 (#3129) Woosuk KwonandGitHub 2024-03-01 12:58:06 -08:00
• c0c2335ce0 Integrate Marlin Kernels for Int4 GPTQ inference (#2497) 4 people 2024-03-01 14:47:51 -06:00
• 90fbf12540 fix relative import path of protocol.py (#3134) 3 people 2024-03-02 03:42:06 +08:00
• 49d849b3ab docs: Add tutorial on deploying vLLM model with KServe (#2586) Yuan TangandGitHub 2024-03-01 14:04:14 -05:00
• 27ca23dc00 Remove exclude_unset in streaming response (#3143) SeonghyeonandGitHub 2024-03-02 02:59:06 +09:00
• 54d3544784 Fix: Output text is always truncated in some models (#3016) SherryandGitHub 2024-03-01 15:52:22 +08:00
• 703e42ee4b Add guided decoding for OpenAI API server (#2819) 4 people 2024-02-29 14:13:08 -08:00
• 29a8d6a554 [Fix] Don't deep-copy LogitsProcessors when copying SamplingParams (#3099) Nick HillandGitHub 2024-02-29 11:20:42 -08:00
• 2c08ff23c0 Fix building from source on WSL (#3112) Billy CaoandGitHub 2024-03-01 03:13:58 +08:00
• bfdcfa6a05 Support starcoder2 architecture (#3089) SeonghyeonandGitHub 2024-02-29 17:51:48 +09:00
• 9289e577ec add cache_config's info to prometheus metrics. (#3100) Allen.DouandGitHub 2024-02-29 14:15:18 +08:00
• a6d471c759 Fix: AttributeError in OpenAI-compatible server (#3018) Jae-Won ChungandGitHub 2024-02-29 01:04:07 -05:00
• 01a5d18a53 Add Support for 2/3/8-bit GPTQ Quantization Models (#2330) CHU TianxiangandGitHub 2024-02-29 13:52:23 +08:00
• 929b4f2973 Add LoRA support for Gemma (#3050) Woosuk KwonandGitHub 2024-02-28 13:03:28 -08:00
• 3b7178cfa4 [Neuron] Support inference with transformers-neuronx (#2569) Liangfu ChenandGitHub 2024-02-28 09:34:34 -08:00
• e46fa5d52e Restrict prometheus_client >= 0.18.0 to prevent errors when importing pkgs (#3070) Allen.DouandGitHub 2024-02-28 13:38:26 +08:00
• a8683102cc multi-lora documentation fix (#3064) Ganesh JagadeesanandGitHub 2024-02-28 00:26:15 -05:00
• 71bcaf99e2 Enable GQA support in the prefix prefill kernels (#3007) Tao HeandGitHub 2024-02-27 17:14:31 +08:00
• 8b430d7dea [Minor] Fix StableLMEpochForCausalLM -> StableLmForCausalLM (#3046) Woosuk KwonandGitHub 2024-02-26 20:23:50 -08:00
• e0ade06d63 Support logit bias for OpenAI API (#3027) Dylan HawkandGitHub 2024-02-26 19:51:53 -08:00
• 4bd18ec0c7 [Minor] Fix type annotation in fused moe (#3045) Woosuk KwonandGitHub 2024-02-26 19:44:29 -08:00
• 2410e320b3 fix get_ip error in pure ipv6 environment (#2931) JingruandGitHub 2024-02-27 11:22:16 +08:00
• 48a8f4a7fd Support Orion model (#2539) 4 people 2024-02-27 11:17:06 +08:00