Commit Graph

Select branches

Hide Pull Requests

main

Mono Color

• cf29b7eda4 [ROCm][Hardware][AMD][Doc] Documentation update for ROCm (#4376) Hongxia YangandWoosukKwon 2024-04-25 21:12:25 -04:00
• efffb63f58 [Core] Move function tracing setup to util function (#4352) Nick Hill 2024-04-25 16:45:12 -07:00
• 15e7c675b0 [Core] Add shutdown() method to ExecutorBase (#4349) Nick Hill 2024-04-25 16:32:48 -07:00
• b6dcb4d442 [Misc] Fix flash attention backend log (#4368) Roy 2024-04-26 03:43:32 +08:00
• b5b4a398a7 [Mypy] Typing lora folder (#4337) SangBin Cho 2024-04-26 04:13:50 +09:00
• f4bc4de1b1 [Core]refactor aqlm quant ops (#4351) Kunshang Ji 2024-04-25 19:03:56 +00:00
• bd7a8eef25 [Doc] README Phi-3 name fix. (#4372) Caio MendesandCaio Mendes 2024-04-25 14:32:00 -03:00
• 7ee82bef1e [CI/Build] Adding functionality to reset the node's GPUs before processing. (#4213) Alexei-V-Ivanov-AMD 2024-04-25 11:37:20 -05:00
• fbf152d976 [Bugfix][Model] Refactor OLMo model to support new HF format in transformers 4.40.0 (#4324) Isotr0pyandWoosuk Kwon 2024-04-26 00:35:56 +08:00
• 479d69fad0 [Core] Move ray_utils.py from engine to executor package (#4347) Nick Hill 2024-04-24 23:52:22 -07:00
• 96e90fdeb3 [Model] Adds Phi-3 support (#4298) Caio Mendes 2024-04-25 00:06:57 -03:00
• a395a638c2 [Misc] Use public API in benchmark_throughput (#4300) zifeitong 2024-04-24 14:10:24 -07:00
• 2768884ac4 [Doc] Add note for docker user (#4340) youkaichaoandSimon Mo 2024-04-24 14:09:44 -07:00
• aae08249ac [Bugfix] Fix marlin kernel crash on H100 (#4218) alexm-nm 2024-04-24 13:35:01 -04:00
• 7923dcad12 [Misc] Update ShareGPT Dataset Sampling in Serving Benchmark (#4279) Roger Wang 2024-04-24 09:49:13 -07:00
• 3cd9b5bb2d [Core][Distributed] use existing torch.cuda.device (#4318) youkaichao 2024-04-24 09:00:20 -07:00
• 468d761b32 [Misc] Reduce supported Punica dtypes (#4304) Woosuk Kwon 2024-04-23 18:54:33 -07:00
• e4bf860a54 [CI][Build] change pynvml to nvidia-ml-py (#4302) youkaichao 2024-04-23 18:33:12 -07:00
• 91f50a6fe2 [Core][Distributed] use cpu/gloo to initialize pynccl (#4248) youkaichao 2024-04-23 18:32:19 -07:00
• 79a268c4ab [BUG] fixed fp8 conflict with aqlm (#4307) Robert Shaw 2024-04-23 21:26:33 -04:00
• eace8bf0b9 [Kernel] FP8 support for MoE kernel / Mixtral (#4244) Philipp Moritz 2024-04-23 18:18:23 -07:00
• 1e8f4252aa [Bugfix][Frontend] Raise exception when file-like chat template fails to be opened (#4292) Cyrus Leung 2024-04-24 02:19:03 +08:00
• 2b7949c1c2 AQLM CUDA support (#3287) James Flemingandmgoin 2024-04-23 13:59:33 -04:00
• 62b5166bd4 [CI] Add ccache for wheel builds job (#4281) Simon Mo 2024-04-23 09:51:41 -07:00
• d86285a4a4 [Core][Logging] Add last frame information for better debugging (#4278) youkaichao 2024-04-23 09:45:52 -07:00
• d87f39e9a9 [Bugfix] Add init_cached_hf_modules to RayWorkerWrapper (#4286) DefTruth 2024-04-24 00:28:35 +08:00
• d3c8180ac4 [Bugfix] Fixing max token error message for openai compatible server (#4016) Jack Gordley 2024-04-23 12:06:29 +01:00
• 62b8aebc6f [Speculative decoding 7/9] Speculative decoding end-to-end correctness tests. (#3951) Cade Daniel 2024-04-23 01:02:36 -07:00
• 050f285ff6 [Core] Scheduling optimization 2 (#4280) SangBin Cho 2024-04-23 17:02:11 +09:00
• 8f2ea22bde [Core] Some simplification of WorkerWrapper changes (#4183) Nick Hill 2024-04-23 00:49:08 -07:00
• 0ae11f78ab [Mypy] Part 3 fix typing for nested directories for most of directory (#4161) SangBin Cho 2024-04-23 13:32:44 +09:00
• 34128a697e Fix autodoc directives (#4272) Harry MellorandHarry Mellor 2024-04-23 02:53:01 +01:00
• c1b4e4157c [Core][Distributed] use absolute path for library file (#4271) youkaichao 2024-04-22 17:21:48 -07:00
• ceaf4ed003 [Doc] Update the SkyPilot doc with serving and Llama-3 (#4276) Zhanghao Wu 2024-04-22 15:34:31 -07:00
• ad8d696a99 [Core] Scheduler perf fix (#4270) SangBin Cho 2024-04-23 06:11:06 +09:00
• 3d925165f2 Add example scripts to documentation (#4225) Harry MellorandHarry Mellor 2024-04-22 17:36:54 +01:00
• 1543680691 [Bugfix] Ensure download_weights_from_hf(..) inside loader is using the revision parameter (#4217) alexm-nm 2024-04-22 12:10:48 -04:00
• 077f0a2e8a [Frontend] Enable support for CPU backend in AsyncLLMEngine. (#3993) Tao He 2024-04-22 17:19:51 +08:00
• e73ed0f1c6 [Bugfix] Fix type annotations in CPU model runner (#4256) Woosuk Kwon 2024-04-22 00:54:16 -07:00
• 296cdf8ac7 [Misc] Add vision language model support to CPU backend (#3968) Isotr0py 2024-04-22 15:44:16 +08:00
• 747b1a7147 [Core][Distributed] fix _is_full_nvlink detection (#4233) youkaichao 2024-04-21 23:04:16 -07:00
• 95e5b087cf [AMD][Hardware][Misc][Bugfix] xformer cleanup and light navi logic and CI fixes and refactoring (#4129) Hongxia Yang 2024-04-22 00:57:24 -04:00
• a37d815b83 Make initialization of tokenizer and detokenizer optional (#3748) 3 people 2024-04-21 15:06:46 -07:00
• 7f2593b164 [Doc]: Update the doc of adding new models (#4236) xiaoji 2024-04-22 00:57:08 +08:00
• fe7d648fe5 Don't show default value for flags in EngineArgs (#4223) Harry MellorandHarry Mellor 2024-04-21 17:15:28 +01:00
• cc74b2b232 Updating lm-format-enforcer version and adding links to decoding libraries in docs (#4222) Noam Gat 2024-04-20 11:33:16 +03:00
• 91528575ec [Frontend] multiple sampling params support (#3570) nunjunj 2024-04-20 00:11:57 -07:00
• a22cdea371 [Kernel][FP8] Initial support with dynamic per-tensor scaling (#4118) Cody Yu 2024-04-19 21:28:57 -07:00
• 682789d402 Fix missing docs and out of sync EngineArgs (#4219) Harry MellorandHarry Mellor 2024-04-20 04:51:33 +01:00
• 138485a82d [Bugfix] Add fix for JSON whitespace (#4189) Ayush RautwarandUbuntu 2024-04-19 23:49:22 -04:00
• bc9df1571b Pass tokenizer_revision when getting tokenizer in openai serving (#4214) Chirag Jain 2024-04-20 05:43:56 +05:30
• 15b86408a8 [Misc] add nccl in collect env (#4211) youkaichao 2024-04-19 12:44:51 -07:00
• 7be4f5628f [Bugfix][Core] Restore logging of stats in the async engine (#4150) Ronen Schaffer 2024-04-19 18:08:26 +03:00
• 8f20fc04bf [Misc] fix docstrings (#4191) UranusandZhong Wang 2024-04-19 16:18:33 +08:00
• 221d93ecbf Bump version of 0.4.1 (#4177) Simon Mo 2024-04-19 01:00:22 -07:00
• d17c8477f1 [Bugfix] Fix LoRA loading check (#4138) Jee Liandsimon-mo 2024-04-19 15:59:54 +08:00
• a134ef6f5e Support eos_token_id from generation_config.json (#4182) Simon Mo 2024-04-18 21:13:36 -07:00
• 8a7a3e4436 [Core] add an option to log every function call to for debugging hang/crash in distributed inference (#4079) youkaichaoandSimon Mo 2024-04-18 16:15:12 -07:00
• 8f9c28fd40 [Bugfix] Fix CustomAllreduce nvlink topology detection (#3974) Adam Tilghman 2024-04-18 15:32:47 -07:00
• cd2f63fb36 [CI/CD] add neuron docker and ci test scripts (#3571) Liangfu Chen 2024-04-18 15:26:01 -07:00
• 87fa80c91f [Misc] Bump transformers to latest version (#4176) Nick Hill 2024-04-18 14:36:39 -07:00
• e1bb2fd52d [Bugfix] Support logprobs when using guided_json and other constrained decoding fields (#4149) James Whedbee 2024-04-18 16:12:55 -05:00
• 705578ae14 [Docs] document that Meta Llama 3 is supported (#4175) Simon Mo 2024-04-18 10:55:48 -07:00
• e8cc7967ff [Bugfix][Kernel] allow non-power-of-two head sizes in prefix prefill (#4128) Michał Moskal 2024-04-18 00:51:28 -07:00
• 53b018edcb [Bugfix] Get available quantization methods from quantization registry (#4098) Michael Goin 2024-04-18 03:21:55 -04:00
• 66ded03067 Allow model to be served under multiple names (#2894) Harry MellorandAlexandre Payot 2024-04-18 08:16:26 +01:00
• 6dc1fc9cfe [Core] nccl integrity check and test (#4155) youkaichao 2024-04-17 22:28:52 -07:00
• 533d2a1f39 [Typing] Mypy typing part 2 (#4043) SangBin ChoandSangBin Cho 2024-04-18 09:28:43 +09:00
• a53222544c [Kernel] Add punica dimension for Swallow-MS-7B LoRA (#4134) Shoichi Uchinami 2024-04-18 02:02:45 +09:00
• fe3b5bbc23 [Bugfix] fix output parsing error for trtllm backend (#4137) ElinxandRoger Wang 2024-04-17 19:07:23 +08:00
• 8438e0569e [Core] RayWorkerVllm --> WorkerWrapper to reduce duplication (#4024) youkaichao 2024-04-17 01:34:33 -07:00
• 11d652bd4f [CI] Move CPU/AMD tests to after wait (#4123) Cade Daniel 2024-04-16 22:53:26 -07:00
• d150e4f89f [Misc] [CI] Fix CI failure caught after merge (#4126) Cade Daniel 2024-04-16 17:56:01 -07:00
• e95cd87959 [Speculative decoding 6/9] Integrate speculative decoding with LLMEngine (#3894) Cade Daniel 2024-04-16 13:09:21 -07:00
• 69e1d2fb69 [Core] Refactor model loading code (#4097) Antoni Baum 2024-04-16 11:34:39 -07:00
• 05434764cd LM Format Enforcer Guided Decoding Support (#3868) Noam GatandSimon Mo 2024-04-16 08:54:57 +03:00
• 4e7ee664e2 [Core] Fix engine-use-ray broken (#4105) SangBin Cho 2024-04-16 14:24:53 +09:00
• 37e84a403d [Typing] Fix Sequence type GenericAlias only available after Python 3.9. (#4092) SangBin Cho 2024-04-16 06:47:31 +09:00
• 4695397dcf [Bugfix] Fix ray workers profiling with nsight (#4095) Ricky Xu 2024-04-15 14:24:45 -07:00
• d619ae2d19 [Doc] Add better clarity for tensorizer usage (#4090) Sanger SteelandRoger Wang 2024-04-15 16:28:25 -04:00
• eb46fbfda2 [Core] Simplifications to executor classes (#4071) Nick Hill 2024-04-15 13:05:09 -07:00
• 0003e9154b [Misc][Minor] Fix CPU block num log in CPUExecutor. (#4088) Li, Jiang 2024-04-15 23:35:55 +08:00
• e11e200736 [Bugfix] Fix filelock version requirement (#4075) Zhuohan Li 2024-04-14 21:50:08 -07:00
• 8db1bf32f8 [Misc] Upgrade triton to 2.2.0 (#4061) Roy 2024-04-15 08:43:54 +08:00
• aceb17cf2d [Docs] document that mixtral 8x22b is supported (#4073) Simon Mo 2024-04-14 14:35:55 -07:00
• 563c54f760 [BugFix] Fix tensorizer extra in setup.py (#4072) Nick Hill 2024-04-14 22:12:42 +01:00
• 2cd6b4f362 [Core] avoid too many cuda context by caching p2p test (#4021) youkaichao 2024-04-13 23:40:21 -07:00
• 711a000255 [Frontend] [Core] feat: Add model loading using tensorizer (#3476) Sanger Steel 2024-04-13 20:13:01 -04:00
• 989ae2538d [Kernel] Add punica dimension for Baichuan-13B (#4053) Jee Li 2024-04-13 22:55:05 +08:00
• 0a430b4ae2 [Bugfix] fix_small_bug_in_neuron_executor (#4051) zspo 2024-04-13 22:54:03 +08:00
• ec8e3c695f [Bugfix] fix_log_time_in_metrics (#4050) zspo 2024-04-13 22:52:36 +08:00
• 98afde19fc [Core][Distributed] improve logging for init dist (#4042) youkaichao 2024-04-13 07:12:53 -07:00
• 5c2e66e487 [Bugfix] More type hint fixes for py 3.8 (#4039) Dylan Hawk 2024-04-12 21:07:04 -07:00
• 546e721168 [CI/Test] expand ruff and yapf for all supported python version (#4037) youkaichao 2024-04-12 18:43:37 -07:00
• b8aacac31a [Bugfix] Fix LoRA bug (#4032) Jee Li 2024-04-13 07:56:37 +08:00
• d04973ad54 Fix triton compilation issue (#3984) Bellk17andWoosuk Kwon 2024-04-12 16:41:26 -07:00
• fbb9d9eef4 [Core] fix custom allreduce default value (#4040) youkaichao 2024-04-12 16:40:39 -07:00
• 09473ee41c [mypy] Add mypy type annotation part 1 (#4006) SangBin Cho 2024-04-13 06:35:50 +09:00
• d4ec9ffb95 [Misc] Fix typo in scheduler.py (#4022) Zhuohan Li 2024-04-12 13:56:04 -07:00
• 96b6a6d790 [Bugfix] fix type hint for py 3.8 (#4036) youkaichao 2024-04-12 12:35:44 -07:00