wassname/vllm
1
0 Fork 0
mirror of https://github.com/wassname/vllm.git synced 2026-10-29 13:50:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Doc] neuron documentation update (#8671)

Browse Source 
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
This commit is contained in:
omrishiv authored and GitHub committed 2024-09-20 15:04:37 -07:00
1 parent b4e4eda92e
commit 7c8566aa4f
2 files changed
+3 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/source/getting_started/neuron-installation.rst
+2 -2
View File
@@ -3,8 +3,8 @@
Installation with Neuron
========================
vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK.
At the moment Paged Attention is not supported in Neuron SDK, but naive continuous batching is supported in transformers-neuronx.
vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK with continuous batching.
Paged Attention and Chunked Prefill are currently in development and will be available soon.
Data types currently supported in Neuron SDK are FP16 and BF16.
Requirements