wassname/talk
1
0
Fork 0
mirror of https://github.com/wassname/talk.git synced 2026-08-13 12:40:11 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into next-amp-docs

Browse Source
This commit is contained in:
Kim Gardner
2019-05-29 12:59:22 +01:00
committed by GitHub
parent 5580ab385f eff3f1dfd8
commit b99161f0cb
6 changed files with 20 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/integrating/asset-scraping.md
+9 -2
View File
@@ -9,6 +9,9 @@ in a simple way. We use the following
[meta tags](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta) on
the target pages that allow us to extract some properties.
Asset scraping is performed by the `scraper` job which is enabled by default when you launch Talk. If your production site is behind a paywall or otherwise prevents scraping, you might need to confiugre a [TALK_SCRAPER_PROXY_URL](/talk/advanced-configuration/#talk-scraper-proxy-url) or custom [TALK_SCRAPER_HEADERS](/talk/advanced-configuration/#talk-scraper-headers).
| Asset Property | Selector |
|--------------------|----------|
| `title` | See [`metascraper-title`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-title/index.js) |
@@ -19,7 +22,7 @@ the target pages that allow us to extract some properties.
| `modified_date` | `meta[property="article:modified"]` |
| `section` | `meta[property="article:section"]` |
You can use the `./bin/cli assets debug <url>` to print the scraped metadata
You can use the `./bin/cli assets debug <url>` command to print the scraped metadata
from that URL. For example:
```bash
@@ -41,4 +44,8 @@ from that URL. For example:
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ section │ │
└──────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
```
You can use the `./bin/cli assets refresh [age]` to trigger scraping or rescrape assets where the scraper job was unsuccessful.