added streaming worker

2026-09-09 11:15:08 +08:00 · 2023-01-26 16:41:57 +01:00
parent c2fa476904
commit f1edcc8a28
4 changed files with 58 additions and 23 deletions
@@ -26,6 +26,13 @@ pip install -r requirements.txt
 python __main__.py
 ```

+For the worker, you'll also want to have the text-generation-inference server
+running:
+
+```bash
+docker run --rm -it -p 8001:80 -e MODEL_ID=distilgpt2 ykilcher/text-generation-inference
+```
+
 Run the client:

 ```bash