wassname/talk
1
0
Fork 0
mirror of https://github.com/wassname/talk.git synced 2026-07-29 11:28:24 +08:00
Code Issues Packages Projects Releases Wiki Activity

Metadata Scraper Debugger (#2048)

Browse Source 
* feat: added asset scrape debugger

* fix: added docs for scraper

* fix: added asset scraping to sidebar
This commit is contained in:
Wyatt Johnson
2018-10-30 17:02:33 +00:00
committed by GitHub
parent 399f5ef7ba
commit 7d246bba76
8 changed files with 175 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/integrating/asset-scraping.md
+44
View File
@@ -0,0 +1,44 @@
---
title: Asset Scraping
permalink: /integrating/asset-scraping/
---
By default, Assets in Talk have their metadata scraped when they are loaded.
This provides the easiest way for newsrooms to integrate their CMS's into Talk
in a simple way. We use the following
[meta tags](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta) on
the target pages that allow us to extract some properties.
| Asset Property | Selector |
|--------------------|----------|
| `title` | See [`metascraper-title`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-title/index.js) |
| `description` | See [`metascraper-description`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-description/index.js) |
| `image` | See [`metascraper-image`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-image/index.js) |
| `author` | See [`metascraper-author`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-author/index.js) |
| `publication_date` | See [`metascraper-date`](https://github.com/microlinkhq/metascraper/blob/dc664c37ea1b238b1e3e9d5342edfacc9027892c/packages/metascraper-date/index.js) |
| `modified_date` | `meta[property="article:modified"]` |
| `section` | `meta[property="article:section"]` |
You can use the `./bin/cli assets debug <url>` to print the scraped metadata
from that URL. For example:
```bash
$ ./bin/cli assets debug https://www.washingtonpost.com/technology/2018/10/30/apple-event-october-ipad-pro-macbook-air/
┌──────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Property │ Value │
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ title │ Apple redesigns the iPad Pro, breathes new life in the MacBook Air │
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ description │ Apple is unveiling new iPads and MacBooks at an event in New York starting at 10 a.m. Fowler is there and will report in with the news and hands-on analysis throughout the day. │
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ image │ https://www.washingtonpost.com/resizer/JAwNQE2alL2JjiWrbXeJ46wZHqA=/1484x0/arc-anglerfish-washpost-prod-washpost.s3.amazonaws.com/public/G5TWBFW4LAI6RC5MX7QB7TODUY.jpg │
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ author │ Geoffrey A. Fowler │
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ publication_date │ 2018-10-30T10:40:00.000Z │
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ modified_date │ │
├──────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ section │ │
└──────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```