mirror of
https://github.com/wassname/psaw.git
synced 2026-09-09 11:30:51 +08:00
root commit
This commit is contained in:
David Marx
@@ -0,0 +1,22 @@
|
||||
Copyright (c) 2018, David Marx
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are met:
|
||||
|
||||
1. Redistributions of source code must retain the above copyright notice, this
|
||||
list of conditions and the following disclaimer.
|
||||
2. Redistributions in binary form must reproduce the above copyright notice,
|
||||
this list of conditions and the following disclaimer in the documentation
|
||||
and/or other materials provided with the distribution.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
|
||||
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
|
||||
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
||||
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
||||
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
||||
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
||||
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
||||
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
+157
@@ -0,0 +1,157 @@
|
||||
Python Pushshift.io API Wrapper (for comment/submission search)
|
||||
===============================================================
|
||||
|
||||
.. _installation:
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install -e git+git://github.com/dmarx/psaw.git
|
||||
|
||||
At present, only python 3 is supported.
|
||||
|
||||
Description
|
||||
-----------
|
||||
|
||||
A minimalist wrapper for searching public reddit comments/submissions via the pushshift.io API.
|
||||
|
||||
Pushshift is an extremely useful resource, but the API is poorly documented. As such, this API wrapper
|
||||
is currently designed to make it easy to pass pretty much any search parameter the user wants to try.
|
||||
|
||||
Although it is not necessarily reflective of the current status of the API, I recommend you
|
||||
attempt to familiarize yourself with the Pushshift API documentation to better understand what search
|
||||
arguments are likely to work. The documentation is distributed across several locations:
|
||||
|
||||
* `API Documentaion on Google Docs <https://docs.google.com/document/d/171VdjT-QKJi6ul9xYJ4kmiHeC7t_3G31Ce8eozKp3VQ/edit>`_
|
||||
* `API Documentation on github <https://github.com/pushshift/api>`_
|
||||
* `/r/pushshift <https://www.reddit.com/r/pushshift/>`_
|
||||
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
* Handles rate limiting and exponential backoff subject to maximum retries and
|
||||
maximum backoff limits. A minimum rate limit of 1 request per second is used
|
||||
as a default per consultation with Pushshift's maintainer,
|
||||
`/u/Stuck_in_the_matrix <https://www.reddit.com/u/Stuck_in_the_matrix>`_.
|
||||
* Handles paging of results. Returns all historical results for a given query by default.
|
||||
* Returns results in ``comment`` and ``submission`` objects whose API is similar to the corresponding ``praw``
|
||||
objects. Additionally, result objects have an additional ``.d_`` attribute that offers dict
|
||||
access to the associated data attributes.
|
||||
* Adds a ``created`` attribute which converts a comment/submission's ``created_utc`` timestamp
|
||||
to the user's local time.
|
||||
* Extremely simple interface to pass query arguments to the API. The API is sparsely documented,
|
||||
so it's often fruitful to just try an argument and see if it works.
|
||||
* Limited support for pushshift's ``aggs`` argument.
|
||||
|
||||
Demo usage
|
||||
----------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from psaw import PushshiftAPI
|
||||
|
||||
api = PushshiftAPI()
|
||||
|
||||
|
||||
100 most recent submissions
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# The `search_comments` and `search_submissions` methods return generator objects
|
||||
gen = api.search_submissions(limit=100)
|
||||
results = list(gen)
|
||||
|
||||
First 10 submissions to /r/politics in 2017, filtering results to url/author/title/subreddit fields.
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The ``created_utc`` field will be added automatically (it's used for paging).
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
list(api.search_submissions(after=int(dt.datetime(2017, 1, 1).timestamp()),
|
||||
subreddit='politics',
|
||||
filter=['url','author', 'title', 'subreddit'],
|
||||
limit=10))
|
||||
|
||||
Trying a search argument that doesn't actually work
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
According to the pushshift.io API documentation, we should be able to search submissions by url,
|
||||
but (at the time of this writing) this doesn't actually work in practice.
|
||||
The API should still respect the ``limit`` argument and possibly other supported arguments,
|
||||
but no guarantees. If you find that an argument you have passed is not supported by the API,
|
||||
best thing is to just remove it from the query and modify your api call to only utilize
|
||||
supported arguments to mitigate risks from of unexpected behavior.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
url = 'http://www.politico.com/story/2017/02/mike-flynn-russia-ties-investigation-235272'
|
||||
url_results = list(api.search_submissions(url=url, limit=500))
|
||||
|
||||
len(url_results), any(r.url == url for r in url_results)
|
||||
# 500, False
|
||||
|
||||
All AskReddit comments containing the text "OP"
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Use the ``q`` parameter to search text. Omitting the ``limit`` parameter does a full
|
||||
historical search. Requests are performed in batches of size specified by the
|
||||
``max_results_per_request`` parameter (default=500). Omitting the "max_reponse_cache"
|
||||
test in the demo below will return all results. Otherwise, this demo will perform two
|
||||
API requests returning 500 comments each. Alternatively, the generator can be queried for additional results.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
gen = api.search_comments(q='OP', subreddit='askreddit')
|
||||
|
||||
max_response_cache = 1000
|
||||
cache = []
|
||||
|
||||
for c in gen:
|
||||
cache.append(c)
|
||||
|
||||
# Omit this test to actually return all results. Wouldn't recommend it though: could take a while, but you do you.
|
||||
if len(cache) >= max_response_cache:
|
||||
break
|
||||
|
||||
# If you really want to: pick up where we left off to get the rest of the results.
|
||||
if False:
|
||||
for c in gen:
|
||||
cache.append(c)
|
||||
|
||||
Using the ``aggs`` argument to count comments mentioning trump each hour in past week
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
Replicating the example from the pushshift documentation:
|
||||
|
||||
https://api.pushshift.io/reddit/search/comment/?q=trump&after=7d&aggs=created_utc&frequency=hour&size=0
|
||||
|
||||
I haven't really experimented much with this functionality of the API, so I figured
|
||||
the simplest way to support it would be to just disable most of the bells and whistles
|
||||
provided by the API wrapper when the ``aggs`` argument is provided (i.e. paging, converting
|
||||
the result to a namedtuple for dot notation attribute access).
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
api = PushshiftAPI()
|
||||
gen = api.search_comments(q='trump',
|
||||
after='7d',
|
||||
aggs='created_utc',
|
||||
frequency='hour',
|
||||
size=0,
|
||||
)
|
||||
|
||||
result = next(gen)
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
PSAW's source is provided under the `Simplified BSD License
|
||||
<https://github.com/dmarx/psaw/master/LICENSE>`_.
|
||||
|
||||
* Copyright (c), 2018, David Marx
|
||||
@@ -0,0 +1,126 @@
|
||||
from collections import namedtuple
|
||||
import copy
|
||||
import json
|
||||
import requests
|
||||
import time
|
||||
from datetime import datetime as dt
|
||||
|
||||
class PushshiftAPIMinimal(object):
|
||||
base_url = 'https://api.pushshift.io/reddit/{}/search/'
|
||||
_limited_args = ('aggs')
|
||||
def __init__(self,
|
||||
max_retries=20,
|
||||
max_sleep=3600,
|
||||
backoff=2,
|
||||
rate_limit=1,
|
||||
max_results_per_request=500
|
||||
):
|
||||
assert rate_limit >=1
|
||||
assert max_results_per_request <= 500
|
||||
assert backoff >= 1
|
||||
|
||||
self.max_retries = max_retries
|
||||
self.max_sleep = max_sleep
|
||||
self.backoff = backoff
|
||||
self.rate_limit = rate_limit
|
||||
self.max_results_per_request = max_results_per_request
|
||||
self._last_request_time = 0
|
||||
self._utc_offset_secs = None
|
||||
|
||||
@property
|
||||
def utc_offset_secs(self):
|
||||
if not self._utc_offset_secs:
|
||||
#self._utc_offset_secs = dt.datetime.utcnow().astimezone().utcoffset().total_seconds()
|
||||
self._utc_offset_secs = dt.utcnow().astimezone().utcoffset().total_seconds()
|
||||
return self._utc_offset_secs
|
||||
|
||||
def _limited(self, payload):
|
||||
"""Turn off bells and whistles for special API endpoints"""
|
||||
return any(arg in payload for arg in self._limited_args)
|
||||
|
||||
def _epoch_utc_to_local(self, epoch):
|
||||
return epoch - self.utc_offset_secs
|
||||
|
||||
def _wrap_thing(self, thing, kind):
|
||||
"""Mimic praw.Submission and praw.Comment API"""
|
||||
thing['created'] = self._epoch_utc_to_local(thing['created_utc'])
|
||||
thing['d_'] = copy.deepcopy(thing)
|
||||
ThingType = namedtuple(kind, thing.keys())
|
||||
thing = ThingType(**thing)
|
||||
return thing
|
||||
|
||||
def _rate_limit(self, nth_request=0):
|
||||
d = time.time() - self._last_request_time
|
||||
interval = max(self.rate_limit, self.backoff*nth_request)
|
||||
interval = min(interval, self.max_sleep)
|
||||
if d < interval:
|
||||
time.sleep(interval-d)
|
||||
self._last_request_time = time.time()
|
||||
|
||||
def _add_nec_args(self, payload):
|
||||
#if 'aggs' in payload:
|
||||
if self._limited(payload):
|
||||
# Do nothing I guess? Not sure how paging works on this endpoint...
|
||||
return
|
||||
if 'limit' not in payload:
|
||||
payload['limit'] = self.max_results_per_request
|
||||
if 'filter' in payload and payload.get('created_utc', None) is None:
|
||||
if not isinstance(payload['filter'], list):
|
||||
payload['filter'] = list(payload['filter'])
|
||||
payload['filter'].append('created_utc')
|
||||
|
||||
def _get(self, kind, payload):
|
||||
self._add_nec_args(payload)
|
||||
url = self.base_url.format(kind)
|
||||
i, success = 0, False
|
||||
while (not success) and (i<self.max_retries):
|
||||
self._rate_limit(i)
|
||||
response = requests.get(url, params=payload)
|
||||
success = response.status_code == 200
|
||||
i+=1
|
||||
response_json = json.loads(response.text)
|
||||
outv = response_json['data']
|
||||
#if 'aggs' in payload:
|
||||
if self._limited(payload):
|
||||
outv = response_json
|
||||
return outv
|
||||
|
||||
def _query(self, kind, stop_condition=lambda **x: False, **kwargs):
|
||||
limit = kwargs.get('limit', None)
|
||||
payload = copy.deepcopy(kwargs)
|
||||
n = 0
|
||||
while True:
|
||||
if limit is not None:
|
||||
if limit > self.max_results_per_request:
|
||||
payload['limit'] = self.max_results_per_request
|
||||
limit -= self.max_results_per_request
|
||||
else:
|
||||
payload['limit'] = limit
|
||||
limit = 0
|
||||
|
||||
results = self._get(kind, payload)
|
||||
#if 'aggs' in payload:
|
||||
if self._limited(payload):
|
||||
yield results
|
||||
return
|
||||
|
||||
if len(results) == 0:
|
||||
return
|
||||
for thing in results:
|
||||
n+=1
|
||||
if stop_condition(**thing):
|
||||
return
|
||||
thing = self._wrap_thing(thing, kind)
|
||||
yield thing
|
||||
payload['before'] = thing.created_utc
|
||||
if (limit is not None) & (limit == 0):
|
||||
return
|
||||
def search_submissions(self, **kwargs):
|
||||
return self._query(kind='submission', **kwargs)
|
||||
|
||||
def search_comments(self, **kwargs):
|
||||
return self._query(kind='comment', **kwargs)
|
||||
|
||||
class PushshiftAPI(PushshiftAPIMinimal):
|
||||
# Fill out this class with more user-friendly features later
|
||||
pass
|
||||
@@ -0,0 +1,9 @@
|
||||
"""
|
||||
Pushshift.io API Wrapper (for reddit.com public comment/submission search)
|
||||
|
||||
https://github.com/dmarx/psaw
|
||||
"""
|
||||
|
||||
from .PushshiftAPI import PushshiftAPI, PushshiftAPIMinimal
|
||||
|
||||
__version__ = '0.0.1'
|
||||
@@ -0,0 +1,40 @@
|
||||
from setuptools import setup
|
||||
from os import path
|
||||
import re
|
||||
|
||||
PACKAGE_NAME='psaw'
|
||||
HERE = path.abspath(path.dirname(__file__))
|
||||
|
||||
with open(path.join(HERE, 'README.md'), encoding='utf-8') as fp:
|
||||
README = fp.read()
|
||||
|
||||
with open(path.join(HERE, PACKAGE_NAME, '__init__.py'),
|
||||
encoding='utf-8') as fp:
|
||||
VERSION = re.search("__version__ = '([^']+)'", fp.read()).group(1)
|
||||
|
||||
setup(name=PACKAGE_NAME,
|
||||
packages=[PACKAGE_NAME],
|
||||
version=VERSION,
|
||||
long_description=README,
|
||||
description='Pushshift.io API Wrapper for reddit.com public comment/submission search',
|
||||
author='David Marx',
|
||||
author_email='david.marx84@gmail.com',
|
||||
url='http://github.com/dmarx/psaw',
|
||||
license='Simplified BSD License',
|
||||
install_requires=['requests'],
|
||||
classifiers=[
|
||||
'Development Status :: 3 - Alpha',
|
||||
'Environment :: Console',
|
||||
'Intended Audience :: Developers',
|
||||
'License :: OSI Approved :: BSD License',
|
||||
'Natural Language :: English',
|
||||
'Operating System :: OS Independent',
|
||||
'Programming Language :: Python',
|
||||
'Programming Language :: Python :: 3',
|
||||
'Programming Language :: Python :: 3.4',
|
||||
'Programming Language :: Python :: 3.5',
|
||||
'Programming Language :: Python :: 3.6',
|
||||
'Programming Language :: Python :: 3 :: Only',
|
||||
'Programming Language :: Python :: Implementation :: CPython',
|
||||
'Topic :: Utilities']
|
||||
)
|
||||
Reference in New Issue
Block a user