mirror of
https://github.com/wassname/Open-Assistant.git
synced 2026-07-27 11:18:29 +08:00
Richard NagyfiandGitHub
55a52fd61f
* Added Gutenberg Crawler Notebook - added configrable Jupyter Notebook with crawler + ability to save to parquet - added README.md - added requirements.txt for running in venv / collab
93 lines
3.1 KiB
Markdown
93 lines
3.1 KiB
Markdown
---
|
|
dataset_info:
|
|
features:
|
|
- name: Text#
|
|
dtype: int64
|
|
- name: Issued
|
|
dtype: timestamp[us]
|
|
- name: Title
|
|
dtype: string
|
|
- name: Authors
|
|
dtype: string
|
|
- name: Subjects
|
|
dtype: string
|
|
- name: LoCC
|
|
dtype: string
|
|
- name: Bookshelves
|
|
dtype: string
|
|
- name: Body
|
|
dtype: string
|
|
splits:
|
|
- name: train
|
|
num_bytes: 193811664
|
|
num_examples: 495
|
|
download_size: 125691249
|
|
dataset_size: 193811664
|
|
license: mit
|
|
task_categories:
|
|
- text-generation
|
|
- conversational
|
|
language:
|
|
- hu
|
|
tags:
|
|
- project gutenberg
|
|
- ebook
|
|
- gutenberg.org
|
|
pretty_name: Hungarian langauge eBooks from Project Gutenberg
|
|
size_categories:
|
|
- n<1K
|
|
---
|
|
|
|
# Dataset Card for "gutenberg_hu"
|
|
|
|
# Dataset Card for Project Gutenberg - Hungarian eBooks
|
|
|
|
## Dataset Description
|
|
|
|
- **Repository:**
|
|
[Code](https://github.com/LAION-AI/Open-Assistant/openassistant/datasets/gutenberg/)
|
|
|
|
## Source data
|
|
|
|
Please **READ** the site's TOS before running the crawler Notebook and follow
|
|
these instructions:
|
|
|
|
- The website will IP ban crawlers for going through each book's metadata page
|
|
separately. Instead use `catalog()` to access the list of available E-books.
|
|
For more information, visit: https://www.gutenberg.org/ebooks/feeds.html
|
|
- You can avoid running the crawler by mirroring the entire database of Project
|
|
Gutenberg or use one of their FTPs instead, and then call the `parse()`
|
|
function on each text
|
|
- For more on robot access see:
|
|
https://www.gutenberg.org/policy/robot_access.html
|
|
|
|
How does it work?
|
|
|
|
- The crawler downloads the raw HTML code for each E-book based on **Text#** id
|
|
(if available)
|
|
- The metadata and the body of text are not clearly separated so the parser will
|
|
try to split them, then remove transcriber's notes and e-book related
|
|
information from the body of text (text marked as copyrighted or malformed
|
|
will be skipped)
|
|
- If there is text both the metadata and the cleared body of text are saved, the
|
|
latter is then added to a filtered parquet file (will contain only the catalog
|
|
information and body of text for the books that were successfully retrieved)
|
|
|
|
Copyright notice:
|
|
|
|
- Some of the books are copyrighted! The crawler (parser) will ignore all books
|
|
with an english copyright header by utilizing a regex expression, but make
|
|
sure to check out the metadata for each book manually to ensure they are okay
|
|
to use in your country! More information on copyright:
|
|
https://www.gutenberg.org/help/copyright.html and
|
|
https://www.gutenberg.org/policy/permission.html
|
|
- Project Gutenberg has the following requests when using books without
|
|
metadata: _Books obtianed from the Project Gutenberg site should have the
|
|
following legal note next to them: "This eBook is for the use of anyone
|
|
anywhere in the United States and most other parts of the world at no cost and
|
|
with almost" no restrictions whatsoever. You may copy it, give it away or
|
|
re-use it under the terms of the Project Gutenberg License included with this
|
|
eBook or online at www.gutenberg.org. If you are not located in the United
|
|
States, you will have to check the laws of the country where you are located
|
|
before using this eBook."_
|