wassname/vllm
1
0
Fork 0
mirror of https://github.com/wassname/vllm.git synced 2026-09-11 12:51:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

[ Kernel ] FP8 Dynamic Per Token Quant - Add scale_ub (#6593)

Browse Source 
Co-authored-by: Varun Sundar Rabindranth <varun@neuralmagic.com>
This commit is contained in:
Varun Sundar Rabindranath
2024-07-19 18:15:26 -07:00
committed by GitHub
co-authored by Varun Sundar Rabindranth
parent e81522e879
commit 2e26564259
6 changed files with 86 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
csrc/torch_bindings.cpp
+1 -1
View File
@@ -188,7 +188,7 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
// Compute dynamic-per-token FP8 quantized tensor and scaling factor.
ops.def(
"dynamic_per_token_scaled_fp8_quant(Tensor! out, Tensor input, Tensor! "
"scale) -> "
"scale, Tensor? scale_ub) -> "
"()");
ops.impl("dynamic_per_token_scaled_fp8_quant", torch::kCUDA,
&dynamic_per_token_scaled_fp8_quant);