Compare commits

...
Author SHA1 Message Date
Jim Miller 38ab66907b Bump versions. 2014-01-13 19:05:36 -06:00
Jim Miller 1dedde2f9d Fix for hpfanficarchive.com changes. 2014-01-13 18:42:36 -06:00
Jim Miller fc5dd20853 Updated French translation. 2014-01-08 17:05:07 -06:00
Jim Miller 4e9fe9f86f First version of French translation. 2014-01-08 14:56:09 -06:00
Jim Miller 7325be98f5 davidfor's changes for storiesonline.net universe/series. 2014-01-05 18:30:58 -06:00
Jim Miller e2545a0da8 Encoding, series & universe for storiesonline.net. 2014-01-05 10:06:03 -06:00
Jim Miller 5cd3dba4a6 Default utf8 first for finestories.com. 2014-01-04 12:11:47 -06:00
Jim Miller bc2420d143 Fix for 'update' with no updates and 'mark' enabled. 2014-01-04 12:11:13 -06:00
Jim Miller 099e8a62ca Added tag FanFictionDownLoader-4.4.89 for changeset d8f5d11b860f 2013-12-23 17:49:48 -05:00
Jim Miller 7f9e22e1b9 Update index and CLI download zip. 2013-12-23 17:49:28 -05:00
Jim Miller a46532bc72 Added tag calibre-plugin-1.8.06 for changeset dd7269e25be8 2013-12-23 17:46:44 -05:00
Jim Miller f7429140f8 Bump versions. 2013-12-23 17:45:19 -05:00
Jim Miller 7872b25034 Minor tweak to ffnet fix in case they change it back, sleep on ffnet meta lookup. 2013-12-23 17:45:08 -05:00
Jim Miller 3d468a4dc5 New site: literotica.com, tweaks to htmlheuristics for it. 2013-12-23 17:43:32 -05:00
asbjorn grandt 413525a881 url for fictionratings changed to https, breaking the soup find. 2013-12-23 08:14:10 +01:00
Jim Miller c066a809c7 Added tag FanFictionDownLoader-4.4.88 for changeset 7ed8deaec5d2 2013-12-12 18:10:51 -06:00
Jim Miller 443b2b5ef8 Save CLI download version *4.4.88* 2013-12-12 18:10:35 -06:00
Jim Miller 3e7f05ba44 Save CLI download version 4.4.87 2013-12-12 18:09:22 -06:00
Jim Miller 25e65da239 Added tag calibre-plugin-1.8.05 for changeset 8254e19f1ab2 2013-12-12 18:03:35 -06:00
Jim Miller c5f211f49f Moved tag calibre-plugin-1.8.04 to changeset 6e301aeb7f4b (from changeset e45187f4215e) 2013-12-12 18:03:26 -06:00
Jim Miller 20fff58813 Moved tag calibre-plugin-1.8.03 to changeset ea92e962bcaa (from changeset 6e301aeb7f4b) 2013-12-12 18:03:00 -06:00
Jim Miller 8fe250747c Added tag calibre-plugin-1.8.04 for changeset e45187f4215e 2013-12-12 17:59:56 -06:00
Jim Miller 4d41c28f3c Fix a bug with AO3 single chapter stories introduced with do_update_hook. 2013-12-12 17:59:41 -06:00
Jim Miller f8d1b21007 Moved tag FanFictionDownLoader-4.4.87 to changeset 8d78c6f57151 (from changeset 6e301aeb7f4b) 2013-12-12 13:35:55 -06:00
Jim Miller 72861e6cd5 Save CLI Zip file. 2013-12-12 13:35:30 -06:00
Jim Miller bd9dba5813 Added tag FanFictionDownLoader-4.4.87 for changeset 6e301aeb7f4b 2013-12-12 13:33:46 -06:00
Jim Miller 1478df572e Added tag calibre-plugin-1.8.03 for changeset 6e301aeb7f4b 2013-12-12 13:33:30 -06:00
Jim Miller 68466af944 Bump versions. 2013-12-12 13:33:13 -06:00
Jim Miller 7c9c84256c Graceful fail for 'k'HTML when no 'k'Url, bump min calibre version. 2013-12-11 12:14:55 -06:00
Jim Miller fcbfe96c64 Add do_update_hook to AO3 for updated chapters in the middle, trekiverse alt URL,
reorder defaults.ini files to put user/pass is_adult first in all sections.
2013-12-11 12:13:22 -06:00
Jim Miller e47748dbbc Added tag calibre-plugin-1.8.02 for changeset a277cb866377 2013-12-05 13:16:45 -06:00
Jim Miller 91a6c668d1 Added tag FanFictionDownLoader-4.4.86 for changeset a277cb866377 2013-12-05 13:16:22 -06:00
19 changed files with 2626 additions and 444 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
# ffd-retief-hrd fanfictiondownloader
application: fanfictiondownloader
version: 4-4-86
version: 4-4-90
runtime: python27
api_version: 1
threadsafe: true
+2 -2
View File
@@ -42,8 +42,8 @@ class FanFictionDownLoaderBase(InterfaceActionBase):
description = _('UI plugin to download FanFiction stories from various sites.')
supported_platforms = ['windows', 'osx', 'linux']
author = 'Jim Miller'
version = (1, 8, 02)
minimum_calibre_version = (0, 8, 57)
version = (1, 8, 07)
minimum_calibre_version = (1, 13, 0)
#: This field defines the GUI plugin class that contains all the code
#: that actually does something. Its format is module_path:class_name
+8 -7
View File
@@ -1393,7 +1393,10 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
if prefs['mark'] or (prefs['errorcol'] != '' and prefs['errorcol'] in custom_columns):
self.previous = self.gui.library_view.currentIndex() # used by update_books_finish.
self.gui.status_bar.show_message(_('Adding/Updating %s BAD books.')%len(book_list))
label = custom_columns[prefs['errorcol']]['label']
if (prefs['errorcol'] != '' and prefs['errorcol'] in custom_columns):
label = custom_columns[prefs['errorcol']]['label']
else:
label = None
LoopProgressDialog(self.gui,
book_list,
partial(self.update_error_column_loop, db=self.gui.current_db, label=label),
@@ -1402,12 +1405,10 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
win_title=_("Update calibre for BAD FanFiction stories"),
status_prefix=_("Updated"))
def update_error_column_loop(self,book,db=None,label='errorcol'):
if book['calibre_id']:
custom_columns = self.gui.library_view.model().custom_columns
if (prefs['errorcol'] != '' and prefs['errorcol'] in custom_columns):
logger.debug("add/update bad %s %s %s"%(book['title'],book['url'],book['comment']))
db.set_custom(book['calibre_id'], book['comment'], label=label, commit=True)
def update_error_column_loop(self,book,db=None,label=None):
if book['calibre_id'] and label:
logger.debug("add/update bad %s %s %s"%(book['title'],book['url'],book['comment']))
db.set_custom(book['calibre_id'], book['comment'], label=label, commit=True)
def add_book_or_update_format(self,book,options,prefs,mi=None):
db = self.gui.current_db
File diff suppressed because it is too large Load Diff
+197 -175
View File
@@ -538,6 +538,13 @@ extratags: FanFiction,Testing,HTML
## personal.ini, not defaults.ini.
#is_adult:true
## archiveofourown.org stories allow chapters to be added out of
## order. So the newest chapter may not be the last one. FFDL update
## doesn't like that. If do_update_hook is uncommented and set true,
## the adapter will discard all existing chapters from the newest one
## on when updating to enforce accurate chapters.
#do_update_hook:false
## AO3 adapter defines a few extra metadata entries.
## If there's ever more than 4 series, add series04,series04Url etc.
extra_valid_entries:fandoms,freeformtags,freefromtags,ao3categories,comments,kudos,hits,bookmarks,collections,series00,series01,series02,series03,series00Url,series01Url,series02Url,series03Url
@@ -562,11 +569,6 @@ include_in_freefromtags:freeformtags
#extra_subject_tags:fandoms,freeformtags,ao3categories
[ashwinder.sycophanthex.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape,Hermione Granger
extraships:Severus Snape/Hermione Granger
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -574,6 +576,11 @@ extraships:Severus Snape/Hermione Granger
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape,Hermione Granger
extraships:Severus Snape/Hermione Granger
[asr3.slashzone.org]
## Site dedicated to these categories/characters/ships
extracategories:The Sentinel
@@ -584,9 +591,6 @@ extracategories:The Sentinel
#is_adult:true
[bloodties-fans.com]
## Site dedicated to these categories/characters/ships
extracategories:Blood Ties
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -599,6 +603,9 @@ extracategories:Blood Ties
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Blood Ties
[buffynfaith.net]
## Site dedicated to these categories/characters/ships
extracategories:Buffy: The Vampire Slayer
@@ -609,9 +616,6 @@ extracategories:Buffy: The Vampire Slayer
#is_adult:true
[castlefans.org]
## Site dedicated to these categories/characters/ships
extracategories:Castle
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -624,6 +628,9 @@ extracategories:Castle
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Castle
[chaos.sycophanthex.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
@@ -636,10 +643,23 @@ cover_exclusion_regexp:/images/.*?ribbon.gif
[dark-solace.org]
## Site dedicated to these categories/characters/ships
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#username:YourName
#password:yourpassword
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
#is_adult:true
extracategories:Buffy: The Vampire Slayer
extracharacters:Buffy, Spike
extraships:Spike/Buffy
[dramione.org]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -652,24 +672,11 @@ extraships:Spike/Buffy
## personal.ini, not defaults.ini.
#is_adult:true
[dramione.org]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Draco Malfoy,Hermione Granger
extraships:Draco Malfoy/Hermione Granger
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#username:YourName
#password:yourpassword
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
#is_adult:true
## some sites include images that we don't ever want becoming the
## cover image. This lets you exclude them.
cover_exclusion_regexp:/images/.*?ribbon.gif
@@ -712,22 +719,19 @@ cliches_label:Character Cliches
extracategories:Lord of the Rings
[erosnsappho.sycophanthex.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## some sites include images that we don't ever want becoming the
## cover image. This lets you exclude them.
cover_exclusion_regexp:/images/.*?ribbon.gif
[fanfiction.mugglenet.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
@@ -740,10 +744,10 @@ extracategories:Harry Potter
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[fanfic.potterheadsanonymous.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
@@ -756,16 +760,19 @@ extracategories:Harry Potter
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[fanfiction.portkey.org]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extraships:Harry Potter/Hermione Granger
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[fanfiction.tenhawkpresents.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
@@ -810,18 +817,26 @@ dislikes_label:Dislikes
## Clear FanFiction from defaults, site is original fiction.
extratags:
extra_valid_entries:size,universe,codes
#extra_titlepage_entries:size,universe,codes
extra_valid_entries:size,universe,universeUrl,universeHTML,codes,notice
#extra_titlepage_entries:size,universeHTML,codes,notice
size_label:Size
universe_label:Universe
universeUrl_label:Universe URL
universeHTML_label:Universe
codes_label:Codes
notice_label:Notice
## Assume entryUrl, apply to "<a class='%slink' href='%s'>%s</a>" to
## make entryHTML.
make_linkhtml_entries:universe
## storiesonline.net stories can be in a series or a universe, but not
## both. By default, universe will be populated in 'series' with
## index=0
universe_as_series: true
[grangerenchanted.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Hermione Granger
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -834,6 +849,10 @@ extracharacters:Hermione Granger
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Hermione Granger
## Extra metadata that this adapter knows about. See [dramione.org]
## for examples of how to use them.
extra_valid_entries:read,reviews
@@ -870,18 +889,15 @@ extracharacters:Kirk,Spock
extraships:Kirk/Spock
[lumos.sycophanthex.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[merlinfic.dtwins.co.uk]
## Site dedicated to these categories/characters/ships
extracategories:Merlin
extracategories:Harry Potter
[merlinfic.dtwins.co.uk]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -894,6 +910,9 @@ extracategories:Merlin
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Merlin
[national-library.net]
## Site dedicated to these categories/characters/ships
extracategories:West Wing
@@ -905,24 +924,32 @@ extracategories:NCIS
[netraptor.org]
[nfacommunity.com]
## Site dedicated to these categories/characters/ships
extracategories:NCIS
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:NCIS
[nha.magical-worlds.us]
## Site dedicated to these categories/characters/ships
extracategories:Buffy: The Vampire Slayer
extracharacters:Willow
[occlumency.sycophanthex.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape
[onedirectionfanfiction.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -930,10 +957,15 @@ extracharacters:Severus Snape
#username:YourName
#password:yourpassword
[onedirectionfanfiction.com]
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:One Direction
[pommedesang.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -946,40 +978,27 @@ extracategories:One Direction
## personal.ini, not defaults.ini.
#is_adult:true
[pommedesang.com]
## Site dedicated to these categories/characters/ships
extracategories:Anita Blake Vampire Hunter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#username:YourName
#password:yourpassword
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
[ponyfictionarchive.net]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[ponyfictionarchive.net]
## Site dedicated to these categories/characters/ships
extracategories:My Little Pony: Friendship is Magic
[pretendercentre.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[pretendercentre.com]
## Site dedicated to these categories/characters/ships
extracategories:The Pretender
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[samdean.archive.nu]
## Site dedicated to these categories/characters/ships
extracategories:Supernatural
@@ -987,9 +1006,6 @@ extracharacters:Sam,Dean
extraships:Sam/Dean
[scarhead.net]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1002,6 +1018,9 @@ extracategories:Harry Potter
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[sg1-heliopolis.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
@@ -1032,9 +1051,6 @@ extracharacters:Draco Malfoy,Harry Potter
extraships:Harry Potter/Draco Malfoy
[thehookupzone.net]
## Site dedicated to these categories/characters/ships
extracategories:Criminal Minds
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1047,6 +1063,9 @@ extracategories:Criminal Minds
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Criminal Minds
[themasque.net]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
@@ -1061,9 +1080,6 @@ extracategories:Criminal Minds
#is_adult:true
[thequidditchpitch.org]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1076,19 +1092,19 @@ extracategories:Harry Potter
## personal.ini, not defaults.ini.
#is_adult:true
[tokra.fandomnet.com]
## Site dedicated to these categories/characters/ships
extracategories:Stargate: SG-1
extracategories:Harry Potter
[tokra.fandomnet.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[trekiverse.org]
## Site dedicated to these categories/characters/ships
extracategories:Star Trek
extracategories:Stargate: SG-1
[trekiverse.org]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1101,26 +1117,24 @@ extracategories:Star Trek
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Star Trek
extra_valid_entries:awards
awards_label:Awards
cover_exclusion_regexp:art/.*Awards.jpg
[www.adastrafanfic.com]
## Site dedicated to these categories/characters/ships
extracategories:Star Trek
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[www.dracoandginny.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Draco Malfoy,Ginny Weasley
extraships:Draco Malfoy/Ginny Weasley
extracategories:Star Trek
[www.dracoandginny.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1133,14 +1147,16 @@ extraships:Draco Malfoy/Ginny Weasley
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Draco Malfoy,Ginny Weasley
extraships:Draco Malfoy/Ginny Weasley
[www.thealphagate.com]
## Site dedicated to these categories/characters/ships
extracategories:Stargate: SG-1
[www.checkmated.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1148,6 +1164,9 @@ extracategories:Harry Potter
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[www.destinysgateway.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
@@ -1155,11 +1174,6 @@ extracategories:Harry Potter
#is_adult:true
[www.dokuga.com]
## Site dedicated to these categories/characters/ships
extracategories:InuYasha
extracharacters:Sesshoumaru,Kagome
extraships:Sesshoumaru/Kagome
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1167,6 +1181,11 @@ extraships:Sesshoumaru/Kagome
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:InuYasha
extracharacters:Sesshoumaru,Kagome
extraships:Sesshoumaru/Kagome
[www.dotmoon.net]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
@@ -1222,14 +1241,14 @@ dateUpdated_format:%%Y-%%m-%%d %%H:%%M:%%S
[www.ficbook.net]
[www.fictionalley.org]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## fictionally.org storyIds are not unique. Combine with authorId.
output_filename: ${title}-${siteabbrev}_${authorId}_${storyId}${formatext}
@@ -1255,6 +1274,23 @@ extra_valid_entries:reviews,favs,follows
#password:yourpassword
[www.fimfiction.net]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## fimfiction.net stories can be locked requiring individual
## passwords. If fail_on_password is set, the downloader will fail
## when a password is required rather than prompting every time.
#fail_on_password: false
## fimfiction.net stories allow chapters to be added out of order. So
## the newest chapter may not be the last one. FFDL update doesn't
## like that. If do_update_hook is uncommented and set true, the
## adapter will discard all existing chapters from the newest one on
## when updating to enforce accurate chapters.
#do_update_hook:false
## Site dedicated to these categories/characters/ships
extracategories:My Little Pony: Friendship is Magic
@@ -1285,55 +1321,33 @@ keep_in_order_groupsUrl:true
## make entryHTML.
make_linkhtml_entries:prequel,sequels,groups
[www.harrypotterfanfiction.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## fimfiction.net stories can be locked requiring individual
## passwords. If fail_on_password is set, the downloader will fail
## when a password is required rather than prompting every time.
#fail_on_password: false
## fimfiction.net stories allow chapters to be added out of order. So
## the newest chapter may not be the last one. FFDL update doesn't
## like that. If do_update_hook is uncommented and set true, the
## adapter will discard all existing chapters from the newest one on
## when updating to enforce accurate chapters.
#do_update_hook:false
[www.harrypotterfanfiction.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[www.henneth-annun.net]
## Site dedicated to these categories/characters/ships
extracategories:The Hobbit
[www.hpfandom.net]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[www.hpfanficarchive.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[www.ik-eternal.net]
## Site dedicated to these categories/characters/ships
extracategories:InuYasha
extracharacters:InuYasha,Kagome
extraships:InuYasha/Kagome
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1346,30 +1360,35 @@ extraships:InuYasha/Kagome
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:InuYasha
extracharacters:InuYasha,Kagome
extraships:InuYasha/Kagome
[www.jlaunlimited.com]
## Site dedicated to these categories/characters/ships
extracategories:JLA
[www.libraryofmoria.com]
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
[www.mediaminer.org]
[www.midnightwhispers.ca]
## Site dedicated to these categories/characters/ships
extracategories:Queer as Folk
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Queer as Folk
## some sites include images that we don't ever want becoming the
## cover image. This lets you exclude them.
cover_exclusion_regexp:/stories/999/images/.*?_trophy.png
@@ -1384,10 +1403,6 @@ extracategories:CSI
extraships:Nick Stokes/Greg Sanders
[www.phoenixsong.net]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extraships:Harry Potter/Ginny Weasley
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1398,9 +1413,13 @@ extraships:Harry Potter/Ginny Weasley
## phoenixsong.net, oddly, can have high rated chapters (login
## required) in the middle of a lower rated story. Use this to force
## FFDL to always login to phoenixsong.net so those stories download
## correctly.
## correctly. If you have a login, this is recommended.
#force_login:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extraships:Harry Potter/Ginny Weasley
[www.potionsandsnitches.net]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
@@ -1423,17 +1442,15 @@ extracategories:Psych
#is_adult:true
[www.qaf-fic.com]
## Site dedicated to these categories/characters/ships
extracategories:Queer as Folk
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Queer as Folk
[www.restrictedsection.org]
extracategories:Harry Potter
extragenres:Erotica
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1441,20 +1458,21 @@ extragenres:Erotica
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extragenres:Erotica
[www.scarvesandcoffee.net]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Glee
extracharacters:Kurt Hummel,Blaine Anderson
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[www.simplyundeniable.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1462,15 +1480,18 @@ extracategories:Harry Potter
#username:YourName
#password:yourpassword
[www.sinful-desire.org]
## Site dedicated to these categories/characters/ships
extracategories:Supernatural
extracategories:Harry Potter
[www.sinful-desire.org]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Supernatural
[www.siye.co.uk]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
@@ -1478,25 +1499,26 @@ extracharacters:Harry Potter,Ginny Weasley
extraships:Harry Potter/Ginny Weasley
[www.squidge.org/peja]
# www.squidge.org/peja calls it Fandom <shrug>
## www.squidge.org/peja calls it Fandom <shrug>
category_label:Fandom
# Remove numWords -- www.squidge.org/peja word counts are inaccurate
## Remove numWords -- www.squidge.org/peja word counts are inaccurate
titlepage_entries: seriesHTML,category,genre,language,characters,ships,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,site,description
[www.squidge.org/peja:txt]
## Add URLs since there aren't links.
# Remove numWords -- www.squidge.org/peja word counts are inaccurate
## Add URLs since there aren't links and remove numWords --
## www.squidge.org/peja word counts are inaccurate
titlepage_entries: series,seriesUrl,category,genre,language,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,site,storyUrl, authorUrl, description
[www.storiesofarda.com]
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
[www.thepetulantpoetess.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
@@ -1539,9 +1561,6 @@ collect_series: false
extracategories:Twilight
[www.twilighted.net]
## Site dedicated to these categories/characters/ships
extracategories:Twilight
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1549,13 +1568,13 @@ extracategories:Twilight
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Twilight
## twilighted.net (ab)uses series as personal reading lists.
collect_series: false
[www.twiwrite.net]
## Site dedicated to these categories/characters/ships
extracategories:Twilight
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1563,20 +1582,23 @@ extracategories:Twilight
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Twilight
## twiwrite.net (ab)uses series as personal reading lists.
collect_series: false
[www.walkingtheplank.org]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape,Harry Potter
extraships:Severus Snape/Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape,Harry Potter
extraships:Severus Snape/Harry Potter
[www.whofic.com]
[www.wizardtales.net]
@@ -1598,14 +1620,14 @@ extracategories:X-Men Movie
extracharacters:Wolverine,Rogue
[www.wraithbait.com]
## Site dedicated to these categories/characters/ships
extracategories:Stargate: Atlantis
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Stargate: Atlantis
[overrides]
## It may sometimes be useful to override all of the specific format,
## site and site:format sections in your private configuration. For
Binary file not shown.
+1
View File
@@ -122,6 +122,7 @@ import adapter_scarheadnet
import adapter_fictionpadcom
import adapter_storiesonlinenet
import adapter_trekiverseorg
import adapter_literotica
## This bit of complexity allows adapters to be added by just adding
## importing. It eliminates the long if/else clauses we used to need
@@ -77,7 +77,7 @@ class ArchiveOfOurOwnOrgAdapter(BaseSiteAdapter):
# The site domain. Does have www here, if it uses it.
return 'archiveofourown.org'
@classmethod
@classmethod
def getSiteExampleURLs(self):
return "http://"+self.getSiteDomain()+"/works/123456 http://"+self.getSiteDomain()+"/collections/Some_Archive/works/123456 http://"+self.getSiteDomain()+"/works/123456/chapters/78901"
@@ -177,19 +177,28 @@ class ArchiveOfOurOwnOrgAdapter(BaseSiteAdapter):
self.story.addToList('authorUrl','http://'+self.host+a['href'])
self.story.addToList('author',a.text)
newestChapter = None
self.newestChapterNum = None # save for comparing during update.
# Scan all chapters to find the oldest and newest, on AO3 it's
# possible for authors to insert new chapters out-of-order or
# change the dates of earlier ones by editing them--That WILL
# break epub update.
# Find the chapters:
chapters=soup.findAll('a', href=re.compile(r'/works/'+self.story.getMetadata('storyId')+"/chapters/\d+$"))
self.story.setMetadata('numChapters',len(chapters))
logger.debug("numChapters: (%s)"%self.story.getMetadata('numChapters'))
for x in range(0,len(chapters)):
# just in case there's tags, like <i> in chapter titles.
chapter=chapters[x]
if len(chapters)==1:
self.chapterUrls.append((self.story.getMetadata('title'),'http://'+self.host+chapter['href']+addurl))
else:
if len(chapters)==1:
self.chapterUrls.append((self.story.getMetadata('title'),'http://'+self.host+chapters[0]['href']+addurl))
else:
for index, chapter in enumerate(chapters):
# strip just in case there's tags, like <i> in chapter titles.
self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+chapter['href']+addurl))
# (2013-09-21)
date = stripHTML(chapter.findNext('span'))[1:-1]
chapterDate = makeDate(date,self.dateformat)
if newestChapter == None or chapterDate > newestChapter:
newestChapter = chapterDate
self.newestChapterNum = index
a = metasoup.find('blockquote',{'class':'userstuff'})
if a != None:
@@ -299,6 +308,12 @@ class ArchiveOfOurOwnOrgAdapter(BaseSiteAdapter):
self.setSeries(series_name, series_index)
self.story.setMetadata('seriesUrl',series_url)
def hookForUpdates(self,chaptercount):
if self.oldchapters and len(self.oldchapters) > self.newestChapterNum:
print("Existing epub has %s chapters\nNewest chapter is %s. Discarding old chapters from there on."%(len(self.oldchapters), self.newestChapterNum+1))
self.oldchapters = self.oldchapters[:self.newestChapterNum]
return len(self.oldchapters)
# grab the text for an individual chapter.
def getChapterText(self, url):
logger.debug('Getting chapter text from: %s' % url)
@@ -72,6 +72,13 @@ class FanFictionNetSiteAdapter(BaseSiteAdapter):
def getSiteURLPattern(self):
return r"https?://(www|m)?\.fanfiction\.net/s/\d+(/\d+)?(/|/[^/]+)?/?$"
def _fetchUrl(self,url):
time.sleep(1.0) ## ffnet(and, I assume, fpcom) tends to fail
## more if hit too fast. This is in
## additional to what ever the
## slow_down_sleep_time setting is.
return BaseSiteAdapter._fetchUrl(self,url)
def extractChapterUrlsAndMetadata(self):
# fetch the chapter. From that we will get almost all the
@@ -157,7 +164,7 @@ class FanFictionNetSiteAdapter(BaseSiteAdapter):
a = soup.find('a', href='http://www.fictionratings.com/')
a = soup.find('a', href=re.compile(r'https?://www\.fictionratings\.com/'))
rating = a.string
if 'Fiction' in rating: # if rating has 'Fiction ', strip that out for consistency with past.
rating = rating[8:]
@@ -275,7 +282,7 @@ class FanFictionNetSiteAdapter(BaseSiteAdapter):
return
def getChapterText(self, url):
time.sleep(5.0) ## ffnet(and, I assume, fpcom) tends to fail
time.sleep(4.0) ## ffnet(and, I assume, fpcom) tends to fail
## more if hit too fast. This is in
## additional to what ever the
## slow_down_sleep_time setting is.
@@ -1,6 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Fanficdownloader team
# Copyright 2013 Fanficdownloader team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -37,11 +37,6 @@ class FineStoriesComAdapter(BaseSiteAdapter):
def __init__(self, config, url):
BaseSiteAdapter.__init__(self, config, url)
self.decode = ["Windows-1252",
"utf8"] # 1252 is a superset of iso-8859-1.
# Most sites that claim to be
# iso-8859-1 (and some that claim to be
# utf8) are really windows-1252.
self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
self.password = ""
self.is_adult=False
@@ -51,7 +46,6 @@ class FineStoriesComAdapter(BaseSiteAdapter):
if 'storyInfo' in self.story.getMetadata('storyId'):
self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
# normalized story URL.
self._setURL('http://' + self.getSiteDomain() + '/s/storyInfo.php?id='+self.story.getMetadata('storyId'))
@@ -128,23 +128,22 @@ class HPFanficArchiveComAdapter(BaseSiteAdapter):
# <span class="label">Rated:</span> NC-17<br /> etc
labels = soup.findAll('span',{'class':'label'})
for labelspan in labels:
value = labelspan.nextSibling
val = labelspan.nextSibling
value = unicode('')
while val and not defaultGetattr(val,'class') == 'label':
value += unicode(val)
val = val.nextSibling
label = labelspan.string
#print("label:%s\nvalue:%s"%(label,value))
if 'Summary' in label:
## Everything until the next span class='label'
svalue = ""
while not defaultGetattr(value,'class') == 'label':
svalue += str(value)
value = value.nextSibling
self.setDescription(url,svalue)
#self.story.setMetadata('description',stripHTML(svalue))
self.setDescription(url,value)
if 'Rated' in label:
self.story.setMetadata('rating', value)
self.story.setMetadata('rating', stripHTML(value))
if 'Word count' in label:
self.story.setMetadata('numWords', value)
self.story.setMetadata('numWords', stripHTML(value))
if 'Categories' in label:
cats = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=categories'))
@@ -167,7 +166,7 @@ class HPFanficArchiveComAdapter(BaseSiteAdapter):
self.story.addToList('warnings',warning.string)
if 'Completed' in label:
if 'Yes' in value:
if 'Yes' in stripHTML(value):
self.story.setMetadata('status', 'Completed')
else:
self.story.setMetadata('status', 'In-Progress')
@@ -0,0 +1,237 @@
# -*- coding: utf-8 -*-
# Copyright 2013 Fanficdownloader team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import time
import logging
logger = logging.getLogger(__name__)
import re
import urllib2
import urlparse
import time
from .. import BeautifulSoup as bs
from ..htmlcleanup import stripHTML
from .. import exceptions as exceptions
from base_adapter import BaseSiteAdapter, makeDate
class LiteroticaSiteAdapter(BaseSiteAdapter):
def __init__(self, config, url):
BaseSiteAdapter.__init__(self, config, url)
self.decode = ["utf8",
"Windows-1252"] # 1252 is a superset of iso-8859-1.
# Most sites that claim to be
# iso-8859-1 (and some that claim to be
# utf8) are really windows-1252.
self.story.setMetadata('siteabbrev','litero')
# get storyId from url--url validation guarantees query is only sid=1234
self.story.setMetadata('storyId',self.parsedUrl.path.split('/',)[2])
self.origurl = url
if "http://www.i." in self.origurl:
## accept m(mobile)url, but use www.
self.origurl = self.origurl.replace("http://www.i.","http://www.")
# normalized story URL.
self._setURL("http://"+self.getSiteDomain()\
+"/s/"+self.story.getMetadata('storyId'))
# The date format will vary from site to site.
# http://docs.python.org/library/datetime.html#strftime-strptime-behavior
self.dateformat = '%m/%d/%y'
@staticmethod
def getSiteDomain():
return 'www.literotica.com'
@classmethod
def getAcceptDomains(cls):
return ['www.literotica.com', 'www.i.literotica.com']
@classmethod
def getSiteExampleURLs(self):
#return "http://www.literotica.com/s/story-title http://www.literotica.com/stories/showstory.php?id=1234 http://www.i.literotica.com/stories/showstory.php?id=1234"
return "http://www.literotica.com/s/story-title"
def getSiteURLPattern(self):
return r"http://www(\.i)?\.literotica\.com/s/([a-zA-Z0-9_-]+)"
def extractChapterUrlsAndMetadata(self):
if not (self.is_adult or self.getConfig("is_adult")):
raise exceptions.AdultCheckRequired(self.url)
url1 = self.origurl
logger.debug("first page URL: "+url1)
try:
data1 = self._fetchUrl(url1)
soup1 = bs.BeautifulSoup(data1)
except urllib2.HTTPError, e:
if e.code == 404:
raise exceptions.StoryDoesNotExist(url1)
else:
raise e
#strip comments from soup
[comment.extract() for comment in soup1.findAll(text=lambda text:isinstance(text, bs.Comment))]
# author
a = soup1.find("span", "b-story-user-y")
self.story.setMetadata('authorId', urlparse.parse_qs(a.a['href'].split('?')[1])['uid'])
self.story.setMetadata('authorUrl', a.a['href'])
self.story.setMetadata('author', a.text)
# get the author page
try:
dataAuth = self._fetchUrl(a.a['href'])
soupAuth = bs.BeautifulSoup(dataAuth)
except urllib2.HTTPError, e:
if e.code == 404:
raise exceptions.StoryDoesNotExist(a.a['href'])
else:
raise e
storyLink = soupAuth.find('a', href=url1)
if storyLink is not None:
# pull the published date from the author page
# default values from single link. Updated below if multiple chapter.
date = storyLink.parent.parent.findAll('td')[-1].text
self.story.setMetadata('datePublished', makeDate(date, self.dateformat))
self.story.setMetadata('dateUpdated',makeDate(date, self.dateformat))
# find num of pages
# find a "3 Pages:" string on the page and parse it
pgs = soup1.find("span", "b-pager-caption-t r-d45").string.split(' ')[0]
# If there are multiple pages, find and request the last page
if "1" != pgs:
logger.debug("last page number: "+pgs)
try:
data2 = self._fetchUrl(url1, {'page': pgs})
soup2 = bs.BeautifulSoup(data2)
[comment.extract() for comment in soup2.findAll(text=lambda text:isinstance(text, bs.Comment))]
except urllib2.HTTPError, e:
if e.code == 404:
# TODO: Probably should reformat this
raise exceptions.StoryDoesNotExist(url1, {'page': pgs})
else:
raise e
else:
#If we're already on the last page, copy the soup
soup2 = soup1
# parse out the list of chapters
chaps = soup2.find('div', id='b-series')
if chaps: # may be one post only
#self.chapterUrls = [(ch.a.text, ch.a['href']) for ch in chaps.findAll('li')]
# if there are chapters, lets pull them and title from the
# author page because *this* chapter is omitted from the
# list on the last page.
row = storyLink.parent.parent.previousSibling
while row['class'] != 'ser-ttl':
row = row.previousSibling
seriesTitle = stripHTML(row)
if seriesTitle:
# this regex is deliberately greedy. We want to get the biggest match before a ':'
self.story.setMetadata('title', re.match('(.*):[^:]*$', seriesTitle).group(1))
else:
self.story.setMetadata('title', soup1.h1.string)
# now chapter list. Assumed oldest to newest.
self.chapterUrls = []
row = row.nextSibling
self.story.setMetadata('datePublished',makeDate(stripHTML(row.find('td',{'class':'dt'})), self.dateformat))
while row['class'] == 'sl':
# pages include full URLs.
self.chapterUrls.append((row.a.string,row.a['href']))
if not row.nextSibling:
break
row = row.nextSibling
row = row.previousSibling
self.story.setMetadata('dateUpdated',makeDate(stripHTML(row.find('td',{'class':'dt'})), self.dateformat))
else: # if one post only
self.chapterUrls = [(soup1.h1.string, url1)]
self.story.setMetadata('title', soup1.h1.string)
# normalize on first chapter URL.
self._setURL(self.chapterUrls[0][1])
# reset storyId to first chapter.
self.story.setMetadata('storyId',self.parsedUrl.path.split('/',)[2])
self.story.setMetadata('numChapters', len(self.chapterUrls))
self.story.setMetadata('category', soup1.find('div', 'b-breadcrumbs').findAll('a')[1].string)
self.story.setMetadata('description', soup1.find('meta', {'name': 'description'})['content'])
return
def getChapterText(self, url):
logger.debug('Getting chapter text from: %s' % url)
time.sleep(0.5)
data1 = self._fetchUrl(url)
soup1 = bs.BeautifulSoup(data1)
#strip comments from soup
[comment.extract() for comment in soup1.findAll(text=lambda text:isinstance(text, bs.Comment))]
# get story text
story1 = soup1.find('div', 'b-story-body-x').p
story1.name='div'
story1.append('<br>')
storytext = self.utf8FromSoup(url,story1)
# find num pages
pgs = int(soup1.find("span", "b-pager-caption-t r-d45").string.split(' ')[0])
logger.debug("pages: "+str(pgs))
# get all the pages
for i in xrange(2, pgs+1):
try:
logger.debug("fetching page "+str(i))
time.sleep(0.5)
data2 = self._fetchUrl(url, {'page': i})
soup2 = bs.BeautifulSoup(data2)
[comment.extract() for comment in soup2.findAll(text=lambda text:isinstance(text, bs.Comment))]
story2 = soup2.find('div', 'b-story-body-x').p
story2.name='div'
story2.append('<br>')
storytext += self.utf8FromSoup(url,story2)
except urllib2.HTTPError, e:
if e.code == 404:
raise exceptions.StoryDoesNotExist(url)
else:
raise e
return storytext
def getClass():
return LiteroticaSiteAdapter
@@ -37,11 +37,6 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
def __init__(self, config, url):
BaseSiteAdapter.__init__(self, config, url)
self.decode = ["Windows-1252",
"utf8"] # 1252 is a superset of iso-8859-1.
# Most sites that claim to be
# iso-8859-1 (and some that claim to be
# utf8) are really windows-1252.
self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
self.password = ""
self.is_adult=False
@@ -147,6 +142,10 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
a = soup.find('h1')
self.story.setMetadata('title',stripHTML(a))
notice = soup.find('div', {'class' : 'notice'})
if notice:
self.story.setMetadata('notice',unicode(notice))
# Find authorid and URL from... author url.
a = soup.find('a', href=re.compile(r"/a/\w+"))
self.story.setMetadata('authorId',a['href'].split('/')[2])
@@ -185,23 +184,74 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
self.story.setMetadata('size', lc2.findNext('td', {'class' : 'num'}).text)
lc4 = lc2.findNext('td', {'class' : 'lc4'})
desc = lc4.contents[0]
try:
a = lc4.find('a', href=re.compile(r"/library/show_series.php\?id=\d+"))
i = a.parent.text.split('(')[1].split(')')[0]
self.setSeries(stripHTML(a), i)
self.story.setMetadata('seriesUrl','http://'+self.host+a['href'])
if a:
# if there's a number after the series name, series_contents is a two element list:
# [<a href="...">Title</a>, u' (2)']
series_contents = a.parent.contents
i = 0 if len(series_contents) == 1 else series_contents[1].strip(' ()')
seriesUrl = 'http://'+self.host+a['href']
self.story.setMetadata('seriesUrl',seriesUrl)
series_name = stripHTML(a)
logger.debug("Series name= %s" % series_name)
series_soup = bs.BeautifulSoup(self._fetchUrl(seriesUrl))
if series_soup:
logger.debug("Retrieving Series - looking for name")
series_name = series_soup.find('span', {'id' : 'ptitle'}).text.partition('')[0]
logger.debug("Series name: '{0}'".format(series_name))
self.setSeries(series_name, i)
desc = lc4.contents[2]
# Check if series is in a universe
universes_soup = bs.BeautifulSoup(self._fetchUrl(self.story.getMetadata('authorUrl') + "&type=uni"))
# logger.debug("Universe page=", universes_soup)
if universes_soup:
universes = universes_soup.findAll('div', {'class' : 'ser-box'})
logger.debug("Number of Universes: %d" % len(universes))
for universe in universes:
logger.debug("universe.find('a')={0}".format(universe.find('a')))
# The universe id is in an "a" tag that has an id but nothing else. It is the first tag.
# The id is prefixed with the letter "u".
universe_id = universe.find('a')['id'][1:]
logger.debug("universe_id='%s'" % universe_id)
universe_name = universe.find('div', {'class' : 'ser-name'}).text.partition(' ')[2]
logger.debug("universe_name='%s'" % universe_name)
# If there is link to the story, we have the right universe
story_a = universe.find('a', {'href' : '/s/'+self.story.getMetadata('storyId')})
if story_a:
logger.debug("Story is in a series that is in a universe! The universe is '%s'" % universe_name)
self.story.setMetadata("universe", universe_name)
self.story.setMetadata('universeUrl','http://'+self.host+ '/library/universe.php?id=' + universe_id)
break
except:
pass
try:
a = lc4.find('a', href=re.compile(r"/library/universe.php\?id=\d+"))
if a:
self.story.setMetadata("universe",stripHTML(a))
desc = lc4.contents[2]
# Assumed only one universe, but it does have a URL--use universeHTML
universe_name = stripHTML(a)
universeUrl = 'http://'+self.host+a['href']
logger.debug("Retrieving Universe - about to get page")
universe_soup = bs.BeautifulSoup(self._fetchUrl(universeUrl))
logger.debug("Retrieving Universe - have page")
if universe_soup:
logger.debug("Retrieving Universe - looking for name")
universe_name = universe_soup.find('span', {'id' : 'ptitle'}).text.partition(' &mdash;')[0]
logger.debug("Universes name: '{0}'".format(universe_name))
self.story.setMetadata('universeUrl',universeUrl)
logger.debug("Setting universe name: '{0}'".format(universe_name))
self.story.setMetadata('universe',universe_name)
if self.getConfig("universe_as_series"):
self.setSeries(universe_name, 0)
self.story.setMetadata('seriesUrl',universeUrl)
except:
pass
desc = lc4.contents[0]
self.setDescription('http://'+self.host+'/s/'+self.story.getMetadata('storyId'),desc)
for b in lc4.findAll('b'):
@@ -65,12 +65,16 @@ class TrekiverseOrgAdapter(BaseSiteAdapter):
# The site domain.
return 'trekiverse.org'
@classmethod
def getAcceptDomains(cls):
return ['trekiverse.org','efiction.trekiverse.org']
@classmethod
def getSiteExampleURLs(self):
return "http://"+self.getSiteDomain()+"/efiction/viewstory.php?sid=1234"
return "http://"+self.getSiteDomain()+"/efiction/viewstory.php?sid=1234 http://efiction."+self.getSiteDomain()+"/viewstory.php?sid=1234"
def getSiteURLPattern(self):
return r'http://trekiverse\.org/efiction/viewstory\.php\?sid=\d+'
return r'(http://trekiverse\.org/efiction/viewstory\.php\?sid=\d+|http://efiction\.trekiverse\.org/viewstory\.php\?sid=\d+)'
## Login seems to be reasonably standard across eFiction sites.
def needToLoginCheck(self, data):
+22 -22
View File
@@ -141,28 +141,28 @@ def replace_br_with_p(body):
averageLineLength = contentLinesSum/contentLines
logger.debug(u'---')
logger.debug(u'Lines.............: ' + str(len(lines)))
logger.debug(u'contentLines......: ' + str(contentLines))
logger.debug(u'contentLinesSum...: ' + str(contentLinesSum))
logger.debug(u'longestLineLength.: ' + str(longestLineLength))
logger.debug(u'averageLineLength.: ' + str(averageLineLength))
logger.debug(u'Lines.............: ' + unicode(len(lines)))
logger.debug(u'contentLines......: ' + unicode(contentLines))
logger.debug(u'contentLinesSum...: ' + unicode(contentLinesSum))
logger.debug(u'longestLineLength.: ' + unicode(longestLineLength))
logger.debug(u'averageLineLength.: ' + unicode(averageLineLength))
if breaksMaxIndex == len(breaksCount)-1 and breaksMax < 2:
breaksMaxIndex = 0
breaksMax = breaksCount[0]
logger.debug(u'---')
logger.debug(u'breaks 1: ' + str(breaksCount[0]))
logger.debug(u'breaks 2: ' + str(breaksCount[1]))
logger.debug(u'breaks 3: ' + str(breaksCount[2]))
logger.debug(u'breaks 4: ' + str(breaksCount[3]))
logger.debug(u'breaks 5: ' + str(breaksCount[4]))
logger.debug(u'breaks 6: ' + str(breaksCount[5]))
logger.debug(u'breaks 7: ' + str(breaksCount[6]))
logger.debug(u'breaks 8: ' + str(breaksCount[7]))
logger.debug(u'breaks 1: ' + unicode(breaksCount[0]))
logger.debug(u'breaks 2: ' + unicode(breaksCount[1]))
logger.debug(u'breaks 3: ' + unicode(breaksCount[2]))
logger.debug(u'breaks 4: ' + unicode(breaksCount[3]))
logger.debug(u'breaks 5: ' + unicode(breaksCount[4]))
logger.debug(u'breaks 6: ' + unicode(breaksCount[5]))
logger.debug(u'breaks 7: ' + unicode(breaksCount[6]))
logger.debug(u'breaks 8: ' + unicode(breaksCount[7]))
logger.debug(u'----')
logger.debug(u'max found: ' + str(breaksMax))
logger.debug(u'max Index: ' + str(breaksMaxIndex))
logger.debug(u'max found: ' + unicode(breaksMax))
logger.debug(u'max Index: ' + unicode(breaksMaxIndex))
logger.debug(u'----')
if breaksMaxIndex > 0 and breaksCount[0] > breaksMax and averageLineLength < 90:
@@ -173,13 +173,13 @@ def replace_br_with_p(body):
for i in range(len(breaksCount)):
# if i > 0 or breaksMaxIndex == 0:
if i <= breaksMaxIndex:
logger.debug(str(i) + u' <= breaksMaxIndex (' + str(breaksMaxIndex) + u')')
logger.debug(unicode(i) + u' <= breaksMaxIndex (' + unicode(breaksMaxIndex) + u')')
body = breaksRegexp[i].sub(r'\1</p>\n<p>\3', body)
elif i == breaksMaxIndex+1:
logger.debug(str(i) + u' == breaksMaxIndex+1 (' + str(breaksMaxIndex+1) + u')')
logger.debug(unicode(i) + u' == breaksMaxIndex+1 (' + unicode(breaksMaxIndex+1) + u')')
body = breaksRegexp[i].sub(r'\1</p>\n<p><br/></p>\n<p>\3', body)
else:
logger.debug(str(i) + u' > breaksMaxIndex+1 (' + str(breaksMaxIndex+1) + u')')
logger.debug(unicode(i) + u' > breaksMaxIndex+1 (' + unicode(breaksMaxIndex+1) + u')')
body = breaksRegexp[i].sub(r'\1</p>\n<hr />\n<p>\3', body)
body = breaksRegexp[8].sub(r'</p>\n<hr />\n<p>', body)
@@ -230,7 +230,7 @@ def replace_br_with_p(body):
return tag_sanitizer(body)
def is_valid_block(block):
return str(block).find('<') == 0 and str(block).find('<!') != 0
return unicode(block).find('<') == 0 and unicode(block).find('<!') != 0
def soup_up_div(body):
blockTags = ['address', 'blockquote', 'del', 'div', 'dl', 'fieldset', 'form', 'ins', 'noscript', 'ol', 'p', 'pre', 'table', 'ul']
@@ -247,8 +247,8 @@ def soup_up_div(body):
lastElement = 1 # 1 = block, 2 = nested, 3 = invalid
for i in soup.contents[0]:
if str(i).strip().__len__() > 0:
s = str(i)
if unicode(i).strip().__len__() > 0:
s = unicode(i)
if type(i) == bs.Tag:
if i.name in blockTags:
if lastElement > 1:
@@ -308,7 +308,7 @@ def tag_sanitizer(html):
is_closed = is_closed_tag(rTag[0]) or is_comment_tag(rTag[0])
# is_comment = is_comment_tag(rTag[0])
# logger.debug(u'%s > isEnd: %s > isClosed: %s > isComment: %s'%(name, str(is_end), str(is_closed), str(is_comment)))
# logger.debug(u'%s > isEnd: %s > isClosed: %s > isComment: %s'%(name, unicode(is_end), unicode(is_closed), unicode(is_comment)))
# logger.debug(u'> %s%s\n'%(rTag[0], rTag[1]))
if name in blockTags:
+7 -2
View File
@@ -389,9 +389,14 @@ class Story(Configurable):
elif self.getMetadataRaw('series') != None:
self.setMetadata('seriesHTML',self.getMetadataRaw('series'))
# logger.debug("make_linkhtml_entries:%s"%self.getConfig('make_linkhtml_entries'))
for k in self.getConfigList('make_linkhtml_entries'):
# Assuming list, because it has to be site specific and
# they are all lists.
# they are all lists. Bail if kUrl list not the same
# length.
# logger.debug("\nk:%s\nlist:%s\nlistURL:%s"%(k,self.getList(k),self.getList(k+'Url')))
if len(self.getList(k+'Url')) != len(self.getList(k)):
continue
htmllist=[]
for i, v in enumerate(self.getList(k)):
url = self.getList(k+'Url')[i]
@@ -403,7 +408,7 @@ class Story(Configurable):
url=removeAllEntities(url)
v=removeAllEntities(v)
htmllist.append(linkhtml%('author',url,v))
htmllist.append(linkhtml%(k,url,v))
join_string = self.getConfig("join_string_"+k+"HTML",u", ").replace('\s',' ')
self.setMetadata(k+'HTML',join_string.join(htmllist))
+7 -11
View File
@@ -48,13 +48,9 @@
<h3>fanfiction.net / fimfiction.net</h3>
<p>
Fanfiction.net appears to be blocking access from Google
App Engine, which prevents this web service. There's
nothing I can do about it. At the time of writing, the
latest CLI and calibre plugin versions worked.
</p>
<p>It appears that FimFiction.net is also blocking access from Google
App Engine now.
As of Jan 13, 2014, fanfiction.net &amp; fimfiction.net
are working again. I'd ask that users limit the number of
stories they download from those sites, thanks.
</p>
{% if authorized %}
@@ -68,9 +64,9 @@
<h3>Changes:</h3>
<p>
<ul>
<li>New Site: trekiverse.org/efiction/</li>
<li>Fix for higher rated stories with not logged in on tthfanfic.org.</li>
<li>Add prequels/sequels &amp; fix groups site specific metadata for fimfiction.net.</li>
<li>Change default encoding for finestories.com.</li>
<li>Change default encoding and improve metadata for storiesonline.net, thanks davidfor.</li>
<li>Fixes for hpfanficarchive.com changes.</li>
</ul>
</p>
@@ -82,7 +78,7 @@
If you have any problems with this application, please
report them in
the <a href="http://groups.google.com/group/fanfic-downloader">FanFictionDownLoader Google Group</a>. The
<a href="http://4-4-85.fanfictiondownloader.appspot.com">Previous Version</a> is also available for you to use if necessary.
<a href="http://4-4-89.fanfictiondownloader.appspot.com">Previous Version</a> is also available for you to use if necessary.
</p>
<div id='error'>
{{ error_message }}
+197 -175
View File
@@ -507,6 +507,13 @@ extratags: FanFiction,Testing,HTML
## personal.ini, not defaults.ini.
#is_adult:true
## archiveofourown.org stories allow chapters to be added out of
## order. So the newest chapter may not be the last one. FFDL update
## doesn't like that. If do_update_hook is uncommented and set true,
## the adapter will discard all existing chapters from the newest one
## on when updating to enforce accurate chapters.
#do_update_hook:false
## AO3 adapter defines a few extra metadata entries.
## If there's ever more than 4 series, add series04,series04Url etc.
extra_valid_entries:fandoms,freeformtags,freefromtags,ao3categories,comments,kudos,hits,bookmarks,collections,series00,series01,series02,series03,series00Url,series01Url,series02Url,series03Url
@@ -531,11 +538,6 @@ include_in_freefromtags:freeformtags
#extra_subject_tags:fandoms,freeformtags,ao3categories
[ashwinder.sycophanthex.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape,Hermione Granger
extraships:Severus Snape/Hermione Granger
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -543,6 +545,11 @@ extraships:Severus Snape/Hermione Granger
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape,Hermione Granger
extraships:Severus Snape/Hermione Granger
[asr3.slashzone.org]
## Site dedicated to these categories/characters/ships
extracategories:The Sentinel
@@ -553,9 +560,6 @@ extracategories:The Sentinel
#is_adult:true
[bloodties-fans.com]
## Site dedicated to these categories/characters/ships
extracategories:Blood Ties
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -568,6 +572,9 @@ extracategories:Blood Ties
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Blood Ties
[buffynfaith.net]
## Site dedicated to these categories/characters/ships
extracategories:Buffy: The Vampire Slayer
@@ -578,9 +585,6 @@ extracategories:Buffy: The Vampire Slayer
#is_adult:true
[castlefans.org]
## Site dedicated to these categories/characters/ships
extracategories:Castle
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -593,6 +597,9 @@ extracategories:Castle
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Castle
[chaos.sycophanthex.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
@@ -605,10 +612,23 @@ cover_exclusion_regexp:/images/.*?ribbon.gif
[dark-solace.org]
## Site dedicated to these categories/characters/ships
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#username:YourName
#password:yourpassword
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
#is_adult:true
extracategories:Buffy: The Vampire Slayer
extracharacters:Buffy, Spike
extraships:Spike/Buffy
[dramione.org]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -621,24 +641,11 @@ extraships:Spike/Buffy
## personal.ini, not defaults.ini.
#is_adult:true
[dramione.org]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Draco Malfoy,Hermione Granger
extraships:Draco Malfoy/Hermione Granger
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#username:YourName
#password:yourpassword
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
#is_adult:true
## some sites include images that we don't ever want becoming the
## cover image. This lets you exclude them.
cover_exclusion_regexp:/images/.*?ribbon.gif
@@ -697,22 +704,19 @@ cliches_label:Character Cliches
extracategories:Lord of the Rings
[erosnsappho.sycophanthex.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## some sites include images that we don't ever want becoming the
## cover image. This lets you exclude them.
cover_exclusion_regexp:/images/.*?ribbon.gif
[fanfiction.mugglenet.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
@@ -725,10 +729,10 @@ extracategories:Harry Potter
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[fanfic.potterheadsanonymous.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
@@ -741,16 +745,19 @@ extracategories:Harry Potter
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[fanfiction.portkey.org]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extraships:Harry Potter/Hermione Granger
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[fanfiction.tenhawkpresents.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
@@ -795,18 +802,26 @@ dislikes_label:Dislikes
## Clear FanFiction from defaults, site is original fiction.
extratags:
extra_valid_entries:size,universe,codes
#extra_titlepage_entries:size,universe,codes
extra_valid_entries:size,universe,universeUrl,universeHTML,codes,notice
#extra_titlepage_entries:size,universeHTML,codes,notice
size_label:Size
universe_label:Universe
universeUrl_label:Universe URL
universeHTML_label:Universe
codes_label:Codes
notice_label:Notice
## Assume entryUrl, apply to "<a class='%slink' href='%s'>%s</a>" to
## make entryHTML.
make_linkhtml_entries:universe
## storiesonline.net stories can be in a series or a universe, but not
## both. By default, universe will be populated in 'series' with
## index=0
universe_as_series: true
[grangerenchanted.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Hermione Granger
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -819,6 +834,10 @@ extracharacters:Hermione Granger
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Hermione Granger
## Extra metadata that this adapter knows about. See [dramione.org]
## for examples of how to use them.
extra_valid_entries:read,reviews
@@ -855,18 +874,15 @@ extracharacters:Kirk,Spock
extraships:Kirk/Spock
[lumos.sycophanthex.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[merlinfic.dtwins.co.uk]
## Site dedicated to these categories/characters/ships
extracategories:Merlin
extracategories:Harry Potter
[merlinfic.dtwins.co.uk]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -879,6 +895,9 @@ extracategories:Merlin
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Merlin
[national-library.net]
## Site dedicated to these categories/characters/ships
extracategories:West Wing
@@ -890,24 +909,32 @@ extracategories:NCIS
[netraptor.org]
[nfacommunity.com]
## Site dedicated to these categories/characters/ships
extracategories:NCIS
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:NCIS
[nha.magical-worlds.us]
## Site dedicated to these categories/characters/ships
extracategories:Buffy: The Vampire Slayer
extracharacters:Willow
[occlumency.sycophanthex.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape
[onedirectionfanfiction.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -915,10 +942,15 @@ extracharacters:Severus Snape
#username:YourName
#password:yourpassword
[onedirectionfanfiction.com]
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:One Direction
[pommedesang.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -931,40 +963,27 @@ extracategories:One Direction
## personal.ini, not defaults.ini.
#is_adult:true
[pommedesang.com]
## Site dedicated to these categories/characters/ships
extracategories:Anita Blake Vampire Hunter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
#username:YourName
#password:yourpassword
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
[ponyfictionarchive.net]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[ponyfictionarchive.net]
## Site dedicated to these categories/characters/ships
extracategories:My Little Pony: Friendship is Magic
[pretendercentre.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[pretendercentre.com]
## Site dedicated to these categories/characters/ships
extracategories:The Pretender
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[samdean.archive.nu]
## Site dedicated to these categories/characters/ships
extracategories:Supernatural
@@ -972,9 +991,6 @@ extracharacters:Sam,Dean
extraships:Sam/Dean
[scarhead.net]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -987,6 +1003,9 @@ extracategories:Harry Potter
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[sg1-heliopolis.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
@@ -1017,9 +1036,6 @@ extracharacters:Draco Malfoy,Harry Potter
extraships:Harry Potter/Draco Malfoy
[thehookupzone.net]
## Site dedicated to these categories/characters/ships
extracategories:Criminal Minds
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1032,6 +1048,9 @@ extracategories:Criminal Minds
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Criminal Minds
[themasque.net]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
@@ -1046,9 +1065,6 @@ extracategories:Criminal Minds
#is_adult:true
[thequidditchpitch.org]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1061,19 +1077,19 @@ extracategories:Harry Potter
## personal.ini, not defaults.ini.
#is_adult:true
[tokra.fandomnet.com]
## Site dedicated to these categories/characters/ships
extracategories:Stargate: SG-1
extracategories:Harry Potter
[tokra.fandomnet.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[trekiverse.org]
## Site dedicated to these categories/characters/ships
extracategories:Star Trek
extracategories:Stargate: SG-1
[trekiverse.org]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1086,26 +1102,24 @@ extracategories:Star Trek
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Star Trek
extra_valid_entries:awards
awards_label:Awards
cover_exclusion_regexp:art/.*Awards.jpg
[www.adastrafanfic.com]
## Site dedicated to these categories/characters/ships
extracategories:Star Trek
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[www.dracoandginny.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Draco Malfoy,Ginny Weasley
extraships:Draco Malfoy/Ginny Weasley
extracategories:Star Trek
[www.dracoandginny.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1118,14 +1132,16 @@ extraships:Draco Malfoy/Ginny Weasley
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Draco Malfoy,Ginny Weasley
extraships:Draco Malfoy/Ginny Weasley
[www.thealphagate.com]
## Site dedicated to these categories/characters/ships
extracategories:Stargate: SG-1
[www.checkmated.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1133,6 +1149,9 @@ extracategories:Harry Potter
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[www.destinysgateway.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
@@ -1140,11 +1159,6 @@ extracategories:Harry Potter
#is_adult:true
[www.dokuga.com]
## Site dedicated to these categories/characters/ships
extracategories:InuYasha
extracharacters:Sesshoumaru,Kagome
extraships:Sesshoumaru/Kagome
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1152,6 +1166,11 @@ extraships:Sesshoumaru/Kagome
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:InuYasha
extracharacters:Sesshoumaru,Kagome
extraships:Sesshoumaru/Kagome
[www.dotmoon.net]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
@@ -1207,14 +1226,14 @@ dateUpdated_format:%%Y-%%m-%%d %%H:%%M:%%S
[www.ficbook.net]
[www.fictionalley.org]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## fictionalley.org doesn't have a status metadatum. If uncommented,
## this will be used for status.
#default_value_status:Unknown
@@ -1237,6 +1256,23 @@ extra_valid_entries:reviews,favs,follows
#password:yourpassword
[www.fimfiction.net]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## fimfiction.net stories can be locked requiring individual
## passwords. If fail_on_password is set, the downloader will fail
## when a password is required rather than prompting every time.
#fail_on_password: false
## fimfiction.net stories allow chapters to be added out of order. So
## the newest chapter may not be the last one. FFDL update doesn't
## like that. If do_update_hook is uncommented and set true, the
## adapter will discard all existing chapters from the newest one on
## when updating to enforce accurate chapters.
#do_update_hook:false
## Site dedicated to these categories/characters/ships
extracategories:My Little Pony: Friendship is Magic
@@ -1267,55 +1303,33 @@ keep_in_order_groupsUrl:true
## make entryHTML.
make_linkhtml_entries:prequel,sequels,groups
[www.harrypotterfanfiction.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## fimfiction.net stories can be locked requiring individual
## passwords. If fail_on_password is set, the downloader will fail
## when a password is required rather than prompting every time.
#fail_on_password: false
## fimfiction.net stories allow chapters to be added out of order. So
## the newest chapter may not be the last one. FFDL update doesn't
## like that. If do_update_hook is uncommented and set true, the
## adapter will discard all existing chapters from the newest one on
## when updating to enforce accurate chapters.
#do_update_hook:false
[www.harrypotterfanfiction.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[www.henneth-annun.net]
## Site dedicated to these categories/characters/ships
extracategories:The Hobbit
[www.hpfandom.net]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[www.hpfanficarchive.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
[www.ik-eternal.net]
## Site dedicated to these categories/characters/ships
extracategories:InuYasha
extracharacters:InuYasha,Kagome
extraships:InuYasha/Kagome
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1328,30 +1342,35 @@ extraships:InuYasha/Kagome
## personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:InuYasha
extracharacters:InuYasha,Kagome
extraships:InuYasha/Kagome
[www.jlaunlimited.com]
## Site dedicated to these categories/characters/ships
extracategories:JLA
[www.libraryofmoria.com]
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
[www.mediaminer.org]
[www.midnightwhispers.ca]
## Site dedicated to these categories/characters/ships
extracategories:Queer as Folk
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Queer as Folk
## some sites include images that we don't ever want becoming the
## cover image. This lets you exclude them.
cover_exclusion_regexp:/stories/999/images/.*?_trophy.png
@@ -1366,10 +1385,6 @@ extracategories:CSI
extraships:Nick Stokes/Greg Sanders
[www.phoenixsong.net]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extraships:Harry Potter/Ginny Weasley
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1380,9 +1395,13 @@ extraships:Harry Potter/Ginny Weasley
## phoenixsong.net, oddly, can have high rated chapters (login
## required) in the middle of a lower rated story. Use this to force
## FFDL to always login to phoenixsong.net so those stories download
## correctly.
## correctly. If you have a login, this is recommended.
#force_login:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extraships:Harry Potter/Ginny Weasley
[www.potionsandsnitches.net]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
@@ -1405,17 +1424,15 @@ extracategories:Psych
#is_adult:true
[www.qaf-fic.com]
## Site dedicated to these categories/characters/ships
extracategories:Queer as Folk
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Queer as Folk
[www.restrictedsection.org]
extracategories:Harry Potter
extragenres:Erotica
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1423,20 +1440,21 @@ extragenres:Erotica
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extragenres:Erotica
[www.scarvesandcoffee.net]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Glee
extracharacters:Kurt Hummel,Blaine Anderson
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
[www.simplyundeniable.com]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1444,15 +1462,18 @@ extracategories:Harry Potter
#username:YourName
#password:yourpassword
[www.sinful-desire.org]
## Site dedicated to these categories/characters/ships
extracategories:Supernatural
extracategories:Harry Potter
[www.sinful-desire.org]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Supernatural
[www.siye.co.uk]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
@@ -1460,25 +1481,26 @@ extracharacters:Harry Potter,Ginny Weasley
extraships:Harry Potter/Ginny Weasley
[www.squidge.org/peja]
# www.squidge.org/peja calls it Fandom <shrug>
## www.squidge.org/peja calls it Fandom <shrug>
category_label:Fandom
# Remove numWords -- www.squidge.org/peja word counts are inaccurate
## Remove numWords -- www.squidge.org/peja word counts are inaccurate
titlepage_entries: seriesHTML,category,genre,language,characters,ships,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,site,description
[www.squidge.org/peja:txt]
## Add URLs since there aren't links.
# Remove numWords -- www.squidge.org/peja word counts are inaccurate
## Add URLs since there aren't links and remove numWords --
## www.squidge.org/peja word counts are inaccurate
titlepage_entries: series,seriesUrl,category,genre,language,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,site,storyUrl, authorUrl, description
[www.storiesofarda.com]
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
[www.thepetulantpoetess.com]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
@@ -1521,9 +1543,6 @@ collect_series: false
extracategories:Twilight
[www.twilighted.net]
## Site dedicated to these categories/characters/ships
extracategories:Twilight
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1531,13 +1550,13 @@ extracategories:Twilight
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Twilight
## twilighted.net (ab)uses series as personal reading lists.
collect_series: false
[www.twiwrite.net]
## Site dedicated to these categories/characters/ships
extracategories:Twilight
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
@@ -1545,20 +1564,23 @@ extracategories:Twilight
#username:YourName
#password:yourpassword
## Site dedicated to these categories/characters/ships
extracategories:Twilight
## twiwrite.net (ab)uses series as personal reading lists.
collect_series: false
[www.walkingtheplank.org]
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape,Harry Potter
extraships:Severus Snape/Harry Potter
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Harry Potter
extracharacters:Severus Snape,Harry Potter
extraships:Severus Snape/Harry Potter
[www.whofic.com]
[www.wizardtales.net]
@@ -1580,14 +1602,14 @@ extracategories:X-Men Movie
extracharacters:Wolverine,Rogue
[www.wraithbait.com]
## Site dedicated to these categories/characters/ships
extracategories:Stargate: Atlantis
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Site dedicated to these categories/characters/ships
extracategories:Stargate: Atlantis
[overrides]
## It may sometimes be useful to override all of the specific format,
## site and site:format sections in your private configuration. For
+3 -7
View File
@@ -53,13 +53,9 @@
{% if fic.failure %}
<h3>fanfiction.net / fimfiction.net</h3>
<p>
FYI, fanfiction.net appears to be blocking access from Google
App Engine, which prevents this web service. There's
nothing I can do about it. At the time of writing, the
latest CLI and calibre plugin versions worked.
</p>
<p>It appears that FimFiction.net is also blocking access from Google
App Engine now.
As of Jan 13, 2014, fanfiction.net &amp; fimfiction.net
are working again. I'd ask that users limit the number of
stories they download from those sites, thanks.
</p>
<span id='error'>{{ fic.failure }}</span>
{% endif %}