wassname/stampy-chat
1
0
Fork 0
mirror of https://github.com/wassname/stampy-chat.git synced 2026-09-11 12:50:34 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into main

Browse Source
This commit is contained in:
henri123lemoine
2023-03-24 22:23:33 -04:00
parent c8a33ff114 a845f90412
commit 3e5cfa18f8
1 changed files with 4 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/text_splitter.py
+4 -3
View File
@@ -207,8 +207,9 @@ r
Anyway, at this point we’re getting into specifics of portals, so I’ll cut off the speculation. The point is: if transportation continues to get cheaper and more efficient over time, then we will converge to the world of the portal, or at least something like it. The details do matter - portals are different from teleportation or whatever might actually happen - but any method of fully relaxing transportation constraints will have qualitatively similar results, to a large extent."""
signature = "Title: Humans are very reliable agents, Author: alyssavance, URL: https://www.lesswrong.com/posts/28zsuPaJpKAGSX4zq"
signature = "Title: Humans are very reliable agents, Author: alyssavance"
splitting = TokenSplitter(max_tokens=250, min_tokens=330)
splitting = TokenSplitter(max_tokens=200, min_tokens=300)
blocks = splitting.split(text, signature)
print("\n\n\n".join(blocks))
context = "Context: " + "\n\n---\n\n".join(blocks)
print(context)