Commit Graph

Select branches

Hide Pull Requests

main

Mono Color

• 5288c06aa0 [Kernel] (1/N) Machete - Hopper Optimized Mixed Precision Linear Kernel (#7174) Lucas WilkinsonandGitHub 2024-08-20 09:09:33 -04:00
• b6f99a6ffe [Core] Refactor executor classes for easier inheritance (#7673) Kunshang JiandGitHub 2024-08-20 15:56:50 +08:00
• ad28a74beb [misc][cuda] add warning for pynvml user (#7675) youkaichaoandGitHub 2024-08-20 00:35:09 -07:00
• e6d811dd13 [XPU] fallback to native implementation for xpu custom op (#7670) jianyizhandGitHub 2024-08-20 15:26:09 +08:00
• c4be16e1a7 [misc] add nvidia related library in collect env (#7674) youkaichaoandGitHub 2024-08-19 23:22:49 -07:00
• 3d8a5f063d [CI] Organizing performance benchmark files (#7616) Kuntai DuandGitHub 2024-08-19 22:43:54 -07:00
• f4fc7337bf [Bugfix] support tie_word_embeddings for all models (#5724) Zijian HuandGitHub 2024-08-19 20:00:04 -07:00
• 0df7ec0b2d [ci] Install Buildkite test suite analysis (#7667) Kevin H. LuuandGitHub 2024-08-19 19:55:04 -07:00
• 312f761232 [Speculative Decoding] Fixing hidden states handling in batch expansion (#7508) Abhinav GoyalandGitHub 2024-08-20 06:28:14 +05:30
• e54ebc2f8f [doc] fix doc build error caused by msgspec (#7659) youkaichaoandGitHub 2024-08-19 17:50:59 -07:00
• 67e02fa8a4 [Bugfix] use StoreBoolean instead of type=bool for --disable-logprobs-during-spec-decoding (#7665) Travis JohnsonandGitHub 2024-08-19 18:43:09 -06:00
• 43735bf5e1 [TPU] Remove redundant input tensor cloning (#7660) Woosuk KwonandGitHub 2024-08-19 15:55:04 -07:00
• da115230fd [Bugfix] Don't disable existing loggers (#7664) Andrew SongandGitHub 2024-08-19 15:11:58 -07:00
• 7601cb044d [Core] Support tensor parallelism for GGUF quantization (#7520) Isotr0pyandGitHub 2024-08-20 05:30:14 +08:00
• 47b65a5508 [core] Multi Step Scheduling (#7000) 3 people 2024-08-19 13:52:13 -07:00
• dad961ef5c [Bugfix] fix lora_dtype value type in arg_utils.py - part 2 (#5428) Ali PanahiandGitHub 2024-08-19 13:47:00 -07:00
• 3ac50b47d0 [MISC] Add prefix cache hit rate to metrics (#7606) Cody YuandGitHub 2024-08-19 11:52:07 -07:00
• df845b2b46 [Misc] Remove Gemma RoPE (#7638) Woosuk KwonandGitHub 2024-08-19 09:29:31 -07:00
• 1a36287b89 [Bugfix] Fix xpu build (#7644) Kunshang JiandGitHub 2024-08-19 13:00:09 +08:00
• f710fb5265 [Core] Use flashinfer sampling kernel when available (#7137) 3 people 2024-08-19 11:24:03 +08:00
• ff7ec82c4d [Core] Optimize SPMD architecture with delta + serialization optimization (#7109) SangBin ChoandGitHub 2024-08-18 17:57:20 -07:00
• 200a2ffa6b [Misc] Refactor Llama3 RoPE initialization (#7637) Woosuk KwonandGitHub 2024-08-18 17:18:12 -07:00
• 40e1360bb6 [CI/Build] Add text-only test for Qwen models (#7475) Alex BrooksandGitHub 2024-08-18 17:43:46 -06:00
• e3b318216d [ Bugfix ] Fix Prometheus Metrics With zeromq Frontend (#7279) 3 people 2024-08-18 16:19:48 -04:00
• ab7165f2c7 [TPU] Optimize RoPE forward_native2 (#7636) Woosuk KwonandGitHub 2024-08-18 01:15:10 -07:00
• 0c2fa50b84 [TPU] Use mark_dynamic only for dummy run (#7634) Woosuk KwonandGitHub 2024-08-18 00:18:53 -07:00
• ce143353c6 [TPU] Skip creating empty tensor (#7630) Woosuk KwonandGitHub 2024-08-17 14:22:46 -07:00
• bbf55c4805 [VLM] Refactor MultiModalConfig initialization and profiling (#7530) Roger WangandGitHub 2024-08-17 13:30:55 -07:00
• 1ef13cf92f [Misc]Fix BitAndBytes exception messages (#7626) Jee Jee LiandGitHub 2024-08-18 03:02:14 +08:00
• 832163b875 [ci][test] allow longer wait time for api server (#7629) youkaichaoandGitHub 2024-08-17 11:26:38 -07:00
• e73f76eec6 [Model] Pipeline parallel support for JAIS (#7603) Besher AlkurdiandGitHub 2024-08-17 21:11:09 +03:00
• d95cc0a55c [core][misc] update libcudart finding (#7620) 3 people 2024-08-16 23:01:35 -07:00
• 5bf45db7df [ci][test] fix engine/logger test (#7621) youkaichaoandGitHub 2024-08-16 23:00:59 -07:00
• eed020f673 [misc] use nvml to get consistent device name (#7582) youkaichaoandGitHub 2024-08-16 21:15:13 -07:00
• 7c0b7ea214 [Bugfix] add >= 1.0 constraint for openai dependency (#7612) Xander JohnsonandGitHub 2024-08-16 20:56:01 -07:00
• 4706eb628e [aDAG] Unflake aDAG + PP tests (#7600) SangBin ChoandGitHub 2024-08-16 20:49:30 -07:00
• bae888cb8e [Bugfix] Clear engine reference in AsyncEngineRPCServer (#7618) Rui QiaoandGitHub 2024-08-16 20:44:05 -07:00
• 6bd19551b0 .[Build/CI] Enabling passing AMD tests. (#7610) Alexei-V-Ivanov-AMDandGitHub 2024-08-16 22:25:32 -05:00
• e680349994 [Bugfix] Fix custom_ar support check (#7617) bnellnmandGitHub 2024-08-16 22:05:49 -04:00
• 44f26a9466 [Model] Align nemotron config with final HF state and fix lm-eval-small (#7611) Michael GoinandGitHub 2024-08-16 18:56:34 -04:00
• 37fd47e780 [Kernel] fix types used in aqlm and ggml kernels to support dynamo (#7596) bnellnmandGitHub 2024-08-16 17:00:11 -04:00
• 7759ae958f [Kernel][Misc] dynamo support for ScalarType (#7594) bnellnmandGitHub 2024-08-16 16:59:49 -04:00
• 9f69856356 [Kernel] register punica functions as torch ops (#7591) bnellnmandGitHub 2024-08-16 16:59:38 -04:00
• d4f0f17b02 [Doc] Update quantization supported hardware table (#7595) Michael GoinandGitHub 2024-08-16 16:59:27 -04:00
• b3f4e17935 [Doc] Add docs for llmcompressor INT8 and FP8 checkpoints (#7444) Michael GoinandGitHub 2024-08-16 16:59:16 -04:00
• 93478b63d2 [Core] Fix tracking of model forward time in case of PP>1 (#7440) Mahesh KeralapuraandGitHub 2024-08-16 13:46:01 -07:00
• f366f6339b [spec decode] [4/N] Move update_flash_attn_metadata to attn backend (#7571) 3 people 2024-08-16 11:41:56 -07:00
• 855866caa9 [Kernel] Add tuned triton configs for ExpertsInt8 (#7601) Michael GoinandGitHub 2024-08-16 14:37:01 -04:00
• 7fc23be81c [Kernel] W8A16 Int8 inside FusedMoE (#7415) Mor ZusmanandGitHub 2024-08-16 20:06:51 +03:00
• e837b624f2 [Feature][Hardware][Amd] Add fp8 Linear Layer for Rocm (#7210) Charlie FuandGitHub 2024-08-16 12:06:30 -05:00
• ec724a725e support tqdm in notebooks (#7510) fzyzcjyandGitHub 2024-08-17 00:17:50 +08:00
• 0e39a33c6d [Bugfix][Hardware][AMD][Frontend] add quantization param to embedding checking method (#7513) Gordon WongandGitHub 2024-08-17 00:05:18 +08:00
• 6fc5b0f249 [CI] Fix crashes of performance benchmark (#7500) Kuntai DuandGitHub 2024-08-16 08:08:45 -07:00
• 9587b050fb [Core] Use uvloop with zmq-decoupled front-end (#7570) Nick HillandGitHub 2024-08-15 22:48:07 -07:00
• 54bd9a03c4 register custom op for flash attn and use from torch.ops (#7536) youkaichaoandGitHub 2024-08-15 22:38:56 -07:00
• 50b8d08dbd [Misc/Testing] Use torch.testing.assert_close (#7324) jon-chuangandGitHub 2024-08-15 21:24:04 -07:00
• e165528778 [CI] Move quantization cpu offload tests out of fastcheck (#7574) Michael GoinandGitHub 2024-08-16 00:16:20 -04:00
• 3b19e39dc5 Chat method for offline llm (#5049) 7 people 2024-08-16 09:41:34 +07:00
• 4cd7d47fed [ci/test] rearrange tests and make adag test soft fail (#7572) youkaichaoandGitHub 2024-08-15 19:39:04 -07:00
• f878c8feb0 [Feature]: Add OpenAI server prompt_logprobs support #6508 (#7453) Grant PinkertandGitHub 2024-08-16 12:38:08 +10:00
• b67ae00cdb [Misc] Add quantization config support for speculative model. (#7343) shangmingcandGitHub 2024-08-16 10:34:28 +08:00
• 9c8e2d1161 [Bugfix][Harmless] Fix float16 dtype for model_is_embedding (#7566) Michael GoinandGitHub 2024-08-15 21:26:19 -04:00
• 21313e09e3 [Bugfix] Fix default weight loading for scalars (#7534) Michael GoinandGitHub 2024-08-15 16:10:22 -04:00
• f4da5f7b6d [Misc] Update dockerfile for CPU to cover protobuf installation (#7182) PHILO-HEandGitHub 2024-08-16 01:03:01 +08:00
• 9c1f78d5d6 [Bugfix] update neuron for version > 0.5.0 (#7175) 3 people 2024-08-15 09:44:14 -07:00
• fc93e56143 [Bugfix][TPU] Correct env variable for XLA cache path (#7544) Woosuk KwonandGitHub 2024-08-15 00:02:29 -07:00
• 22b39e11f2 llama_index serving integration documentation (#6973) 3 people 2024-08-15 04:08:37 +05:30
• f55a9aea45 [Misc] Revert compressed-tensors code reuse (#7521) Kyle SayersandGitHub 2024-08-14 18:07:37 -04:00
• 951fdd66d3 [TPU] Set per-rank XLA cache (#7533) Woosuk KwonandGitHub 2024-08-14 14:47:51 -07:00
• 2ecf7b1757 [core] [3/N] multi-step args and sequence.py (#7452) William LinandGitHub 2024-08-14 12:32:45 -07:00
• 3f674a49b5 [VLM][Core] Support profiling with multiple multi-modal inputs per prompt (#7126) Cyrus LeungandGitHub 2024-08-15 01:55:42 +08:00
• 70b746efcf [Misc] Deprecation Warning when setting --engine-use-ray (#7424) 5 people 2024-08-14 13:44:27 -03:00
• 67d115db08 [Bugfix][Frontend] Disable embedding API for chat models (#7504) 3 people 2024-08-15 00:15:19 +08:00
• d3d9cb6e4b [ci] fix model tests (#7507) youkaichaoandGitHub 2024-08-14 01:01:43 -07:00
• c134a46402 Fix empty output when temp is too low (#2937) 3 people 2024-08-13 22:31:44 -07:00
• 199adbb7cf [doc] update test script to include cudagraph (#7501) youkaichaoandGitHub 2024-08-13 21:52:58 -07:00
• dd164d72f3 [Bugfix][Docs] Update list of mock imports (#7493) Cyrus LeungandGitHub 2024-08-14 11:37:30 +08:00
• ea49e6a3c8 [misc][ci] fix cpu test with plugins (#7489) youkaichaoandGitHub 2024-08-13 19:27:46 -07:00
• 97992802f3 [CI/Build]Reduce the time consumption for LoRA tests (#7396) Jee Jee LiandGitHub 2024-08-14 08:27:29 +08:00
• 59edd0f134 [Bugfix][CI] Import ray under guard (#7486) Woosuk KwonandGitHub 2024-08-13 17:12:58 -07:00
• a08df8322e [TPU] Support multi-host inference (#7457) Woosuk KwonandGitHub 2024-08-13 16:31:20 -07:00
• 16422ea76f [misc][plugin] add plugin system implementation (#7426) youkaichaoandGitHub 2024-08-13 16:24:17 -07:00
• 373538f973 [Misc] compressed-tensors code reuse (#7277) Kyle SayersandGitHub 2024-08-13 19:05:15 -04:00
• 33e5d7e6b6 [frontend] spawn engine process from api server process (#7484) youkaichaoandGitHub 2024-08-13 15:40:17 -07:00
• c5c7768264 Announce NVIDIA Meetup (#7483) 3 people 2024-08-13 14:28:36 -07:00
• b1e5afc3e7 [Misc] Update awq and awq_marlin to use vLLMParameters (#7422) Dipika SikkaandGitHub 2024-08-13 17:08:20 -04:00
• d3bdfd3ab9 [Misc] Update Fused MoE weight loading (#7334) Dipika SikkaandGitHub 2024-08-13 14:57:45 -04:00
• fb377d7e74 [Misc] Update gptq_marlin to use new vLLMParameters (#7281) Dipika SikkaandGitHub 2024-08-13 14:30:11 -04:00
• 181abbc27d [Misc] Update LM Eval Tolerance (#7473) Dipika SikkaandGitHub 2024-08-13 14:28:14 -04:00
• 00c3d68e45 [Frontend][Core] Add plumbing to support audio language models (#7446) Peter SalasandGitHub 2024-08-13 10:39:33 -07:00
• e20233d361 Revert "[Doc] Update supported_hardware.rst (#7276)" (#7467) Woosuk KwonandGitHub 2024-08-13 01:37:08 -07:00
• d6e634f3d7 [TPU] Suppress import custom_ops warning (#7458) Woosuk KwonandGitHub 2024-08-13 00:30:30 -07:00
• 4d2dc5072b [hardware] unify usage of is_tpu to current_platform.is_tpu() (#7102) youkaichaoandGitHub 2024-08-13 00:16:42 -07:00
• 7025b11d94 [Bugfix] Fix weight loading for Chameleon when TP>1 (#7410) Cyrus LeungandGitHub 2024-08-13 13:33:41 +08:00
• 5469146bcc [ci] Remove fast check cancel workflow (#7455) Kevin H. LuuandGitHub 2024-08-12 21:19:51 -07:00
• 97a6be95ba [Misc] improve logits processors logging message (#7435) Andrew WangandGitHub 2024-08-12 19:29:34 -07:00
• 9ba85bc152 [mypy] Misc. typing improvements (#7417) Cyrus LeungandGitHub 2024-08-13 09:20:20 +08:00
• 198d6a2898 [Core] Shut down aDAG workers with clean async llm engine exit (#7224) Rui QiaoandGitHub 2024-08-12 17:57:16 -07:00
• 774cd1d3bf [CI/Build] bump minimum cmake version (#6999) DanieleandGitHub 2024-08-13 01:29:20 +02:00
• 91294d56e1 [Bugfix] Handle PackageNotFoundError when checking for xpu version (#7398) sasha0552andGitHub 2024-08-12 23:07:20 +00:00