mirror of
https://github.com/wassname/talk.git
synced 2026-07-20 12:40:47 +08:00
immberandKiwi
73aa77d7da
* updated asset scraping docs * added note about setting close date * added close window setting details
51 lines
9.1 KiB
Markdown
51 lines
9.1 KiB
Markdown
---
|
|
title: Asset Scraping
|
|
permalink: /integrating/asset-scraping/
|
|
---
|
|
|
|
By default, Assets in Talk have their metadata scraped when they are loaded.
|
|
This provides the easiest way for newsrooms to integrate their CMS's into Talk
|
|
in a simple way. We use the following
|
|
[meta tags](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta) on
|
|
the target pages that allow us to extract some properties.
|
|
|
|
Asset scraping is performed by the `scraper` job which is enabled by default when you launch Talk. If your production site is behind a paywall or otherwise prevents scraping, you might need to confiugre a [TALK_SCRAPER_PROXY_URL](/talk/advanced-configuration/#talk-scraper-proxy-url) or custom [TALK_SCRAPER_HEADERS](/talk/advanced-configuration/#talk-scraper-headers).
|
|
|
|
|
|
| Asset Property | Selector |
|
|
|--------------------|----------|
|
|
| `title` | See [`metascraper-title`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-title/index.js) |
|
|
| `description` | See [`metascraper-description`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-description/index.js) |
|
|
| `image` | See [`metascraper-image`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-image/index.js) |
|
|
| `author` | See [`metascraper-author`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-author/index.js) |
|
|
| `publication_date` | See [`metascraper-date`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-date/index.js) |
|
|
| `modified_date` | `meta[property="article:modified"]` |
|
|
| `section` | `meta[property="article:section"]` |
|
|
|
|
You can use the `./bin/cli assets debug <url>` command to print the scraped metadata
|
|
from that URL. For example:
|
|
|
|
```bash
|
|
$ ./bin/cli assets debug https://www.washingtonpost.com/technology/2018/10/30/apple-event-october-ipad-pro-macbook-air/
|
|
┌──────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ Property │ Value │
|
|
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
│ title │ Apple redesigns the iPad Pro, breathes new life in the MacBook Air │
|
|
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
│ description │ Apple is unveiling new iPads and MacBooks at an event in New York starting at 10 a.m. Fowler is there and will report in with the news and hands-on analysis throughout the day. │
|
|
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
│ image │ https://www.washingtonpost.com/resizer/JAwNQE2alL2JjiWrbXeJ46wZHqA=/1484x0/arc-anglerfish-washpost-prod-washpost.s3.amazonaws.com/public/G5TWBFW4LAI6RC5MX7QB7TODUY.jpg │
|
|
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
│ author │ Geoffrey A. Fowler │
|
|
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
│ publication_date │ 2018-10-30T10:40:00.000Z │
|
|
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
│ modified_date │ │
|
|
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
│ section │ │
|
|
└──────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
|
|
|
|
You can use the `./bin/cli assets refresh [age]` to trigger scraping or rescrape assets where the scraper job was unsuccessful. |