henri123lemoine
0fcaeb4a61
add embeddings to chunk table, add stream_chunks
2023-07-17 17:15:07 -04:00
henri123lemoine
d08477b036
clean up code
2023-07-17 17:13:48 -04:00
henri123lemoine
e61d423714
fixed pinecone info, added aisafety.info bias
2023-07-17 17:12:19 -04:00
henri123lemoine
07025fb24f
clean main
2023-07-17 17:06:11 -04:00
henri123lemoine
689a9a7ae4
clean up
2023-07-03 01:02:41 -04:00
henri123lemoine
511f231872
Uncommented pinecone functionality
2023-07-02 07:23:13 -04:00
henri123lemoine
aeba2fbfb0
bug-fix and fixed device option
2023-07-02 07:22:38 -04:00
henri123lemoine
cdfb5a52a7
Added sentence transformer option
2023-07-02 07:09:06 -04:00
henri123lemoine
0cf2fdad74
Implemented batch updating
2023-07-02 06:03:05 -04:00
henri123lemoine
5321af9813
Implemented upsert_chunks in the sql handler
2023-07-02 06:01:31 -04:00
henri123lemoine
f1f0c8a21b
Implemented upsert_entries in the Pinecone handler
2023-07-02 06:01:03 -04:00
henri123lemoine
987d565b11
Added retry to the embeddings method
2023-06-27 11:20:41 -04:00
henri123lemoine
2025940db9
Moving Pinecone env variables assignments to settings
2023-06-27 10:36:15 -04:00
henri123lemoine
084b7139a1
Reformatting
2023-06-27 10:19:41 -04:00
henri123lemoine
0ec3a8f5e4
Reorder imports
2023-06-27 07:50:16 -04:00
henri123lemoine
67f0dafd14
Make env var defs fail-fast
2023-06-27 07:48:18 -04:00
henri123lemoine
49e9f08cf5
Removed magic number 3 by using a settings const
2023-06-27 07:34:14 -04:00
henri123lemoine
11dc2dd6c1
Fixed entry typing and renamed var for clarity
2023-06-27 07:26:41 -04:00
henri123lemoine
560b132e67
Added filenames on line 1 and removed RESET_DBS, fixed logging bug
2023-06-27 07:21:45 -04:00
henri123lemoine
abb506511e
Chaining iterable_data calls
2023-06-27 06:41:21 -04:00
henri123lemoine
0204810aa5
Improving naming
2023-06-27 03:09:51 -04:00
henri123lemoine
5900da48bb
Renamed classes, added settings
2023-06-27 02:45:50 -04:00
henri123lemoine
20346d5fa4
Renamed file
2023-06-27 02:40:00 -04:00
henri123lemoine
473b5ed945
Renamed class and file
2023-06-27 02:39:49 -04:00
henri123lemoine
17ca839d54
Removed logs from gitignore
2023-06-27 02:39:17 -04:00
henri123lemoine
2438a2b01f
Added logger to sql and pinecone handlers
2023-06-27 02:15:35 -04:00
henri123lemoine
f45e0c034f
Removed logging file, added reset_dbs method
2023-06-27 02:06:05 -04:00
henri123lemoine
9314de03e5
Removed time logging and show_stats
2023-06-27 01:54:52 -04:00
henri123lemoine
822d958700
bug fix
2023-06-27 01:54:21 -04:00
henri123lemoine
d9e4cb67e0
Utilized settings file for pinecone handler
2023-06-27 01:50:52 -04:00
henri123lemoine
cda0ce44e9
Simplified update_source method, fixed bugs
2023-06-27 01:48:00 -04:00
henri123lemoine
0c97ea61c5
Added settings file
2023-06-27 01:46:19 -04:00
henri123lemoine
4b64f3bae0
Updated .env.example file
2023-06-27 01:45:59 -04:00
henri123lemoine
049aa315e5
Added ARD_LangChain_QA_Chat notebook to gitignore
2023-06-27 01:45:38 -04:00
henri123lemoine
b25777741d
Added main() for modularity
2023-06-24 00:21:12 -04:00
henri123lemoine
d8c6665a51
Deleted unnecessary lines and added replace_current_index functionality to create_index
2023-06-23 18:59:57 -04:00
henri123lemoine
22b912c814
Added convenient __str__ to the pinecone handler
2023-06-23 18:55:57 -04:00
henri123lemoine
61cb161ffc
Improved validate_entry
2023-06-23 18:48:56 -04:00
henri123lemoine
5ba1e0c22d
Removed unused files
2023-06-23 18:27:14 -04:00
henri123lemoine
1c96ca4112
Updated main.py to call update_dataset.py
2023-06-23 18:27:02 -04:00
henri123lemoine
1d8c025d24
Created update_dataset.py
...
It contains a class, ARDUpdater, that deals with upserting entries to both sql and pinecone databases by checking diffs. It deals with splitting chunks and embedding them before upserting to pinecone.
2023-06-23 18:26:45 -04:00
henri123lemoine
c76ad2df28
Added a Pinecone database handler
2023-06-23 18:23:10 -04:00
henri123lemoine
1514e9ec27
Created an sqlite database handler
2023-06-23 18:22:56 -04:00
henri123lemoine
72c28ad133
added example .env in src/
2023-06-23 18:22:33 -04:00
henri123lemoine
27a56b601e
Updated gitignore
2023-06-23 18:22:18 -04:00
Thomas Lemoine
3b667ce16a
simplified validation
2023-06-23 15:00:50 -04:00
Thomas Lemoine
8044111a1b
commented change in authors' type from str to list
2023-06-21 00:45:40 -04:00
Thomas Lemoine
9b8c470438
removed error_count_dict
2023-06-21 00:41:54 -04:00
Thomas Lemoine
12c9db9cd0
load_dataset simpler than langchain's wrapper
2023-06-21 00:40:26 -04:00
Thomas Lemoine
5479ed7e93
minor change to MAX_LEN_PROMPT
2023-06-21 00:38:43 -04:00