mirror of
https://github.com/wassname/scrape_r_rational.git
synced 2026-09-10 12:35:53 +08:00
92 lines
2.7 KiB
Markdown
92 lines
2.7 KiB
Markdown
# rrational
|
|
|
|
|
|
|
|
scrapping reddit.com/r/rational and analytics
|
|
|
|
- This project has data from r/rational in **markdown** that you can browse, see [./data/cache2/](./data/cache2/) for the data.
|
|
|
|
- This project also has an html **table** you can see/download at https://wassname.github.io/scrape_r_rational/
|
|
|
|
|
|
|
|
|
|
## More info:
|
|
|
|
Reddit Discussion: https://old.reddit.com/r/rational/comments/1hoonrc/v2_table_which_stories_have_been_linked_most/
|
|
|
|
Table Columns
|
|
|
|
- 'Title': LLM's opinion about the title
|
|
- **'⬆️': Sum of comment score for associated links**
|
|
- 'Comments': number of comments with the link in that we found and assocated with this row
|
|
- '⭐Qual': LLM's opinion about the users opinion of quality of the fiction out of 10
|
|
- '⭐Rat': LLM's opinion about the users opinion of the rating of the fiction
|
|
- '⭐Writ': LLM on writing style
|
|
- '⭐Plot': LLM on plot
|
|
- '⭐Char': LLM on characters
|
|
- '⭐World': LLM on wordbuilding
|
|
- **'Tags': LLM's opinion about the tags**
|
|
- 'First Link': Date of the first link
|
|
- 'Last Link': Date of the last link
|
|
- 'Links': Number of associated links
|
|
- 'URLs': List of associated links
|
|
- 'Reviews Summary': An LLM was asked to summarize user reviews
|
|
- **'Threads': Links to all the threads!!**
|
|
- **'Comments': Links to all the comments!!**
|
|
- 'Similar': LLM's opinion about similar fictions
|
|
- 'Description': LLM's description
|
|
- 'Recommendations': LLM on why a r/rational user would reccomend
|
|
- 'Disrecommendations': LLM
|
|
- 'Why': LLM
|
|
- 'Reviews': An LLM was asked to quote user reivews... it made some of them up
|
|
|
|
|
|
You can see the actuall prompt in <nbs/mjc_005_allm.ipynb>, search for `class FictionInfo`
|
|
|
|
|
|
For the Table UI I've included
|
|
- smart search https://datatables.net/reference/option/search.smart
|
|
- search builder https://datatables.net/extensions/searchbuilder/
|
|
- save states
|
|
- export to excel
|
|
- column visibility
|
|
|
|
|
|
## Project plan:
|
|
|
|
- [x] Init
|
|
- [x] Fill out README
|
|
- [x] Scrape r/rational
|
|
- [x] use [statistics](https://github.com/wassname/scrape_r_rational/blob/main/nbs/links.csv)
|
|
- [x] Use llm to get reccomendations, sentiment, karma etc
|
|
- [x] share
|
|
- [x] comment md to html
|
|
- [x] comment expand
|
|
- [x] threads where it's mentioned
|
|
- [x] better tittles and data cleaning
|
|
- [x] github pages
|
|
|
|
|
|
## Install requirements
|
|
|
|
This project uses [poetry](https://python-poetry.org/) for requirement and is set up for torch using cuda.
|
|
~~~
|
|
poetry install
|
|
~~~
|
|
|
|
Then
|
|
~~~
|
|
cp .env.example .env
|
|
~~~
|
|
Then fill out the api keys
|
|
|
|
|
|
## How to run
|
|
|
|
First run <nbs/mjc_001_download.ipynb> to update the data in <data/cache2/>
|
|
|
|
Then run <nbs/mjc_004_process.ipynb> to analyse the data and output <index.html>
|
|
|
|
use <nbs/mjc_005_allm.ipynb> to run an llm from openrouter (costs around $50) and the results are... OK
|