Commit Graph

Select branches

Hide Pull Requests

main

Mono Color

• 3476ed0809 [Core] Optimize block_manager_v2 vs block_manager_v1 (to make V2 default) (#5602) Alexander MatveevandGitHub 2024-07-01 23:10:37 -04:00
• 54600709b6 [Model] Changes to MLPSpeculator to support tie_weights and input_scale (#5965) 3 people 2024-07-02 01:40:02 +02:00
• e373853e12 [Frontend] Relax api url assertion for openai benchmarking (#6046) James WhedbeeandGitHub 2024-07-01 18:39:10 -05:00
• c87ebc3ef9 [BugFix] Ensure worker model loop is always stopped at the right time (#5987) Nick HillandGitHub 2024-07-01 16:17:58 -07:00
• c4059ea54f [Bugfix] Add explicit end_forward calls to flashinfer (#6044) Antoni BaumandGitHub 2024-07-01 16:08:58 -07:00
• 8e0817c262 [Bugfix][Doc] Fix Doc Formatting (#6048) Roger WangandGitHub 2024-07-01 15:09:11 -07:00
• 83bdcb6ac3 add FAQ doc under 'serving' (#5946) ning.zhangandGitHub 2024-07-01 14:11:36 -07:00
• 12a59959ed [Bugfix] adding chunking mechanism to fused_moe to handle large inputs (#6029) Avshalom ManevichandGitHub 2024-07-02 00:08:29 +03:00
• dec6fc6f3b [Bugfix] Use RayActorError for older versions of Ray in RayTokenizerGroupPool (#6039) Antoni BaumandGitHub 2024-07-01 13:12:40 -07:00
• 8893130b63 [doc][misc] further lower visibility of simple api server (#6041) 3 people 2024-07-01 10:50:56 -07:00
• bb60326836 [Misc] update benchmark backend for scalellm (#6018) zhyncsandGitHub 2024-07-02 01:20:33 +08:00
• 4050d646e5 [doc][misc] remove deprecated api server in doc (#6037) youkaichaoandGitHub 2024-07-01 09:52:43 -07:00
• d76084c12f [ CI ] Re-enable Large Model LM Eval (#6031) Robert ShawandGitHub 2024-07-01 12:40:45 -04:00
• 80ca1e6a3a [Speculative Decoding 2/2 ] Integrate typical acceptance sampler into Spec Decode Worker (#5348) sroy745andGitHub 2024-07-01 00:33:05 -07:00
• 614aa51203 [misc][cuda] use nvml to avoid accidentally cuda initialization (#6007) youkaichaoandGitHub 2024-06-30 20:07:34 -07:00
• af9ad46fca [ Misc ] Refactor w8a8 to use process_weights_after_load (Simplify Weight Loading) (#5940) 3 people 2024-06-30 19:06:27 -04:00
• 7836fdcc11 [Misc] Fix get_min_capability (#5971) Dipika SikkaandGitHub 2024-06-30 16:15:16 -04:00
• deacb7ec44 [ CI ] Temporarily Disable Large LM-Eval Tests (#6005) 3 people 2024-06-30 14:56:56 -04:00
• f5e73c9f1b [Lora] Use safetensor keys instead of adapter_config.json to find unexpected modules. (#5909) 3 people 2024-07-01 02:11:15 +09:00
• c6c240aa0a [Frontend]: Support base64 embedding (#5935) 3 people 2024-06-30 08:53:00 -07:00
• 2be6955a3f [ci][distributed] fix device count call youkaichaoandGitHub 2024-06-30 01:06:13 -07:00
• 9d47f64eb6 [CI/Build] [3/3] Reorganize entrypoints tests (#5966) Cyrus LeungandGitHub 2024-06-30 12:58:49 +08:00
• cff6a1fec1 [CI/Build] Reuse code for checking output consistency (#5988) Cyrus LeungandGitHub 2024-06-30 11:44:25 +08:00
• bcc6a09b63 [CI/Build] Temporarily Remove Phi3-Vision from TP Test (#5989) Roger WangandGitHub 2024-06-29 18:18:31 -07:00
• 9def10664e [Bugfix][CI/Build][Hardware][AMD] Install matching torchvision to fix AMD tests (#5949) Matt WongandGitHub 2024-06-29 14:47:58 -05:00
• 75aa1442db [ CI/Build ] LM Eval Harness Based CI Testing (#5838) 3 people 2024-06-29 13:04:30 -04:00
• 99397da534 [CI/Build] Add TP test for vision models (#5892) Cyrus LeungandGitHub 2024-06-29 23:45:54 +08:00
• 8dbfcd35bf [ CI/Build ] Added E2E Test For Compressed Tensors (#5839) 4 people 2024-06-29 09:12:58 -04:00
• f7dac83d95 [Kernel] Raise an exception in MoE kernel if the batch size is larger then 65k (#5939) Cody YuandGitHub 2024-06-29 06:04:20 -07:00
• 7c01f70641 [Core] Optimize SequenceStatus.is_finished by switching to IntEnum (#5974) Antoni BaumandGitHub 2024-06-29 05:47:53 -07:00
• 51e971d39e [Bugfix] Support eos_token_id from config.json (#5954) Cyrus LeungandGitHub 2024-06-29 19:19:02 +08:00
• 329df38f1a [Misc] Update Phi-3-Vision Example (#5981) 3 people 2024-06-28 23:34:29 -07:00
• 580353da93 [Bugfix] Fix precisions in Gemma 1 (#5913) Woosuk KwonandGitHub 2024-06-28 20:10:21 -07:00
• ba4994443a [Kernel] Add punica dimensions for Granite 3b and 8b (#5930) Joe RundeandGitHub 2024-06-28 20:48:25 -06:00
• 906a19cdb0 [Misc] Extend vLLM Metrics logging API (#5925) 3 people 2024-06-28 19:36:06 -07:00
• c4bca740e8 [Bugfix] fix missing last itl in openai completions benchmark (#5926) mcalmanandGitHub 2024-06-28 22:34:42 -04:00
• 7f83f40dee [Bugfix][TPU] Fix pad slot id (#5977) Woosuk KwonandGitHub 2024-06-28 18:55:17 -07:00
• 54814fd85b [Bugfix][TPU] Fix TPU sampler output (#5978) Woosuk KwonandGitHub 2024-06-28 18:14:16 -07:00
• 7041de4384 [Kernel] Flashinfer for prefill & decode, with Cudagraph support for decode (#4628) 3 people 2024-06-28 15:28:49 -07:00
• 6a62cb82cc [Bugfix] Fix Engine Failing After Invalid Request - AsyncEngineDeadError (#5963) 3 people 2024-06-28 17:46:30 -04:00
• 5d2a1a9cf0 Unmark more files as executable (#5962) Tyler Michael SmithandGitHub 2024-06-28 17:34:56 -04:00
• 4bf35ed9ae [Bugfix] Only add Attention.kv_scale if kv cache quantization is enabled (#5936) Michael GoinandGitHub 2024-06-28 17:12:40 -04:00
• be0b3af9e0 Support Deepseek-V2 (#4650) 3 people 2024-06-29 04:24:57 +08:00
• 2cd402e169 [ Bugfix ] Enabling Loading Models With Fused QKV/MLP on Disk with FP8 (#5921) 3 people 2024-06-28 14:43:49 -04:00
• b185230744 [ Misc ] Remove fp8_shard_indexer from Col/Row Parallel Linear (Simplify Weight Loading) (#5928) 3 people 2024-06-28 13:49:57 -04:00
• 6a2d659d28 [Bugfix] Fix compute datatype for cutlass 3.x epilogues (#5931) Tyler Michael SmithandGitHub 2024-06-28 13:10:34 -04:00
• b2c620230a [Spec Decode] Introduce DraftModelRunner (#5799) Cody YuandGitHub 2024-06-28 09:17:51 -07:00
• b90d8cd832 [Distributed] Make it clear that % should not be in tensor dict keys. (#5927) xwjiang2010andGitHub 2024-06-28 08:20:22 -07:00
• 3b752a6555 [CI/Build] [2/3] Reorganize entrypoints tests (#5904) Cyrus LeungandGitHub 2024-06-28 22:59:18 +08:00
• ec1ad0046c [Bugfix] Better error message for MLPSpeculator when num_speculative_tokens is set too high (#5894) Thomas ParnellandGitHub 2024-06-28 16:42:17 +02:00
• 57f09a419c [Hardware][Intel] OpenVINO vLLM backend (#5379) Ilya LavrenovandGitHub 2024-06-28 17:50:16 +04:00
• 5932634409 Unmark fused_moe config json file as executable (#5960) Tyler Michael SmithandGitHub 2024-06-28 09:36:12 -04:00
• 5cbe8d155c [Core] Registry for processing model inputs (#5214) 3 people 2024-06-28 20:09:56 +08:00
• 0d0e3a42ac [Bugfix][Hardware][Intel CPU] Fix unpassed multi_modal_kwargs for CPU runner (#5956) Isotr0pyandGitHub 2024-06-28 20:03:41 +08:00
• 74d55c065b [VLM][BugFix] Make sure that multi_modal_kwargs can broadcast properly with ring buffer. (#5905) 3 people 2024-06-28 00:29:13 -07:00
• f136da15e1 [Hardware][TPU] Optimize KV cache swapping (#5878) Woosuk KwonandGitHub 2024-06-27 21:12:13 -07:00
• c3dde367f1 [Kernel][ROCm][AMD] fused_moe Triton configs v2 for mi300X (#5932) Divakar VermaandGitHub 2024-06-27 15:41:08 -05:00
• 64e8d2a783 [core][misc] remove logical block (#5882) youkaichaoandGitHub 2024-06-27 13:34:55 -07:00
• 79c92c7c8a [Model] Add Gemma 2 (#5908) Woosuk KwonandGitHub 2024-06-27 13:33:56 -07:00
• 736ed38849 [CI/Build] Fix Args for _get_logits_warper in Sampler Test (#5922) Roger WangandGitHub 2024-06-27 11:43:04 -07:00
• 365791ff81 [BugFix] Fix min_tokens behaviour for multiple eos tokens (#5849) Nick HillandGitHub 2024-06-27 11:31:11 -07:00
• 691e29ecf3 [BugFix] Fix MLPSpeculator handling of num_speculative_tokens (#5876) Nick HillandGitHub 2024-06-27 10:59:33 -07:00
• 3fd02bda51 [doc][misc] add note for Kubernetes users (#5916) youkaichaoandGitHub 2024-06-27 10:07:07 -07:00
• 98cf2ed678 [Model][Bugfix] Implicit model flags and reenable Phi-3-Vision (#5896) Cyrus LeungandGitHub 2024-06-28 00:08:10 +08:00
• e9d32d077d [CI/Build] [1/3] Reorganize entrypoints tests (#5526) Cyrus LeungandGitHub 2024-06-27 20:43:17 +08:00
• 2061f0b8a7 [Bugfix] Fix img_sizes Parsing in Phi3-Vision (#5888) Roger WangandGitHub 2024-06-27 01:29:24 -07:00
• 96354d6a29 [Model] Add base class for LoRA-supported models (#5018) Cyrus LeungandGitHub 2024-06-27 16:03:04 +08:00
• d12af207d2 [VLM][Bugfix] Make sure that multi_modal_kwargs is broadcasted properly (#5880) xwjiang2010andGitHub 2024-06-27 00:15:24 -07:00
• 6eabc6cb0e [Doc] Add note about context length in Phi-3-Vision example (#5887) Cyrus LeungandGitHub 2024-06-27 14:20:01 +08:00
• 2110557dab [BugFix] Fix cuda graph for MLPSpeculator (#5875) 3 people 2024-06-26 21:12:10 -07:00
• b9e84259e9 [Misc] Add example for LLaVA-NeXT (#5879) Roger WangandGitHub 2024-06-26 17:57:16 -07:00
• 294104c3f9 [doc] update usage of env var to avoid conflict (#5873) youkaichaoandGitHub 2024-06-26 14:57:12 -07:00
• 38a1674abb Support CPU inference with VSX PowerPC ISA (#5652) Chip KerchnerandGitHub 2024-06-26 17:53:04 -04:00
• f5c8628fdc [Bugfix][TPU] Fix CPU cache allocation (#5869) Woosuk KwonandGitHub 2024-06-26 13:42:40 -07:00
• cbc53b6b8d [Hardware][TPU] Support parallel sampling & Swapping (#5855) Woosuk KwonandGitHub 2024-06-26 11:07:49 -07:00
• c54269d967 [Frontend] Add tokenize/detokenize endpoints (#5054) sasha0552andGitHub 2024-06-26 16:54:22 +00:00
• 5bfd1bbc98 [Kernel] Adding bias epilogue support for cutlass_scaled_mm (#5560) 4 people 2024-06-26 11:16:00 -04:00
• 6984c02a27 [CI/Build] Refactor image test assets (#5821) Cyrus LeungandGitHub 2024-06-26 16:02:34 +08:00
• 3439c5a8e3 [Bugfix][TPU] Fix KV cache size calculation (#5860) Woosuk KwonandGitHub 2024-06-26 00:58:23 -07:00
• 6806998bf9 [Bugfix] Fix embedding to support 2D inputs (#5829) Woosuk KwonandGitHub 2024-06-26 00:15:22 -07:00
• 515080ad2f [bugfix][distributed] fix shm broadcast when the queue size is full (#5801) youkaichaoandGitHub 2024-06-25 21:56:02 -07:00
• 3aa7b6cf66 [Misc][Doc] Add Example of using OpenAI Server with VLM (#5832) Roger WangandGitHub 2024-06-25 20:34:25 -07:00
• dda4811591 [Core] Refactor Worker and ModelRunner to consolidate control plane communication (#5408) 3 people 2024-06-25 20:30:03 -07:00
• 82079729cc [Bugfix] Fix assertion in NeuronExecutor (#5841) aws-patlangeandGitHub 2024-06-25 19:52:10 -07:00
• c2a8ac75e0 [CI/Build] Add E2E tests for MLPSpeculator (#5791) Thomas ParnellandGitHub 2024-06-26 01:04:08 +01:00
• f178e56c68 [Hardware][TPU] Raise errors for unsupported sampling params (#5850) Woosuk KwonandGitHub 2024-06-25 16:58:23 -07:00
• dd793d1de5 [Hardware][AMD][CI/Build][Doc] Upgrade to ROCm 6.1, Dockerfile improvements, test fixes (#5422) Matt WongandGitHub 2024-06-25 17:56:15 -05:00
• bc34937d68 [Hardware][TPU] Refactor TPU backend (#5831) Woosuk KwonandGitHub 2024-06-25 15:25:52 -07:00
• dd248f7675 [Misc] Update w4a16 compressed-tensors support to include w8a16 (#5794) Dipika SikkaandGitHub 2024-06-25 15:23:35 -04:00
• d9b34baedd [CI/Build] Add unit testing for FlexibleArgumentParser (#5798) Michael GoinandGitHub 2024-06-25 15:18:03 -04:00
• c18ebfdd71 [doc][distributed] add both gloo and nccl tests (#5834) youkaichaoandGitHub 2024-06-25 12:10:28 -07:00
• 67882dbb44 [Core] Add fault tolerance for RayTokenizerGroupPool (#5748) Antoni BaumandGitHub 2024-06-25 10:15:10 -07:00
• 7b99314301 [Misc] Remove useless code in cpu_worker (#5824) Jie Fu (傅杰)andGitHub 2024-06-26 00:41:36 +08:00
• 2ce5d6688b [Speculative Decoding] Support draft model on different tensor-parallel size than target model (#5414) Woo-Yeon LeeandGitHub 2024-06-25 18:56:06 +09:00
• f23871e9ee [Doc] Add notice about breaking changes to VLMs (#5818) Cyrus LeungandGitHub 2024-06-25 16:25:03 +08:00
• e9de9dd551 [ci] Remove aws template (#5757) Kevin H. LuuandGitHub 2024-06-24 21:09:02 -07:00
• ba991d5c84 [Bugfix] Fix FlexibleArgumentParser replaces _ with - for actual args (#5795) Chang SuandGitHub 2024-06-24 16:01:19 -07:00
• 1744cc99ba [Doc] Add Phi-3-medium to list of supported models (#5788) Michael GoinandGitHub 2024-06-24 13:48:55 -04:00
• e72dc6cb35 [Doc] Add "Suggest edit" button to doc pages (#5789) Michael GoinandGitHub 2024-06-24 13:26:17 -04:00
• c246212952 [doc][faq] add warning to download models for every nodes (#5783) youkaichaoandGitHub 2024-06-24 00:37:42 -07:00