This commit is contained in:
Matt Apperson
2025-11-13 14:21:27 -05:00
parent d13e797ad1
commit 94ba44c933
795 changed files with 5179 additions and 4513 deletions
+28
View File
@@ -0,0 +1,28 @@
# Tokenizer
Tokenizer type used by the model
## Values
| Name | Value |
| ----------- | ----------- |
| `ROUTER` | Router |
| `MEDIA` | Media |
| `OTHER` | Other |
| `GPT` | GPT |
| `CLAUDE` | Claude |
| `GEMINI` | Gemini |
| `GROK` | Grok |
| `COHERE` | Cohere |
| `NOVA` | Nova |
| `QWEN` | Qwen |
| `YI` | Yi |
| `DEEP_SEEK` | DeepSeek |
| `MISTRAL` | Mistral |
| `LLAMA2` | Llama2 |
| `LLAMA3` | Llama3 |
| `LLAMA4` | Llama4 |
| `PA_LM` | PaLM |
| `RWKV` | RWKV |
| `QWEN3` | Qwen3 |