Compare commits

...
35 Commits
Author SHA1 Message Date
Jim Miller 3d07257f7c Bump version v2.2.4 2015-04-29 11:03:06 -05:00
Jim Miller b7d2fdba81 Update translations. 2015-04-29 11:02:23 -05:00
Jim Miller b2fbe6ea09 Update strings for translation. 2015-04-27 11:56:07 -05:00
Jim Miller 0415f3bd6c Improve Reject list 2015-04-24 13:42:16 -05:00
Jim Miller bad9a0ea36 Merge branch 'master' into calibre_cover_generate 2015-04-24 12:12:37 -05:00
Jim Miller d51bb8373f Fix multi-page chapters on storiesonline.net. 2015-04-24 12:11:39 -05:00
Jim Miller 138b751519 Enhance Cover Generation settings, including option to use Cal's built-in cover gen. 2015-04-23 20:07:34 -05:00
Jim Miller 53e9030a61 Remove 'Include images in EPUBs?' config option from plugin--default to always include images. 2015-04-23 17:56:04 -05:00
Jim Miller 3ab60a9808 Add ini notes about cover_image with generate_cover_settings 2015-04-23 17:46:56 -05:00
Jim Miller bdf5d45a3c Update Calibre Cover settings only--not function yet. 2015-04-23 17:46:07 -05:00
Jim Miller f91aa5ba3f Merge remote-tracking branch 'origin/master' into calibre_cover_generate 2015-04-22 22:27:57 -05:00
Jim Miller 4980dd8027 Fix Spanish(es) %(pini)s--fixed in Transifex 2015-04-22 17:23:54 -05:00
Jim Miller ca6bc7f88e Detect and reject animated gifs in Calibre because they come out so poorly. 2015-04-22 16:50:56 -05:00
Jim Miller 53ca251a49 Add to FimF cover_exclusion_regexp and smaller cover when href not found. 2015-04-22 15:48:24 -05:00
Jim Miller dda8360044 Improve error reporting when only one URL vs list. 2015-04-22 15:40:46 -05:00
Jim Miller f51b21cfeb Add to FimF cover_exclusion_regexp and smaller cover when href not found. 2015-04-22 14:04:25 -05:00
Jim Miller e07bd0653b Add to FimF cover_exclusion_regexp and smaller cover when href not found. 2015-04-22 14:03:58 -05:00
Jim Miller b16aee27a1 More fix for storiesonline.net change 2015-04-21 13:28:20 -05:00
Jim Miller 17a631dc79 More fix for storiesonline.net change 2015-04-21 10:44:49 -05:00
Jim Miller 808b44cf9b Pass allmetadata back out from jobs.py to have cover_image. 2015-04-20 21:19:10 -05:00
Jim Miller 62db815781 Merge branch 'master' into calibre_cover_generate 2015-04-20 21:16:07 -05:00
Jim Miller d2aa6c1774 Save come defaults.ini comment changes. 2015-04-20 21:12:35 -05:00
Jim Miller 48e1bc7aa4 Add cover_image meta-metadata for source of cover image: specific, first, old or default. 2015-04-20 20:32:34 -05:00
Jim Miller f6b1529c27 Add Readme for translation dir 2015-04-20 20:16:02 -05:00
Jim Miller 84b0070d2c Translation updates. 2015-04-18 13:28:31 -05:00
Jim Miller 37e80cb63b Bump version v2.2.3 2015-04-18 13:21:02 -05:00
Jim Miller c82a157bad Add site it-could-happen.net. 2015-04-17 14:09:57 -05:00
Jim Miller 555fa7678f Change code.google link. 2015-04-16 18:09:41 -05:00
Jim Miller eeccde995f Merge branch 'master' of https://github.com/JimmXinu/FanFicFare 2015-04-16 17:11:31 -05:00
Jim Miller e82a21e303 Fix wiki link. 2015-04-16 17:09:03 -05:00
Jim Miller 64d147910d Update README.md 2015-04-16 17:08:10 -05:00
Jim Miller 04e0bde07b Update PI about box. 2015-04-16 13:30:13 -05:00
Jim Miller 0a07ebc8b2 Fix storiesonline.net site changes 2015-04-16 13:24:52 -05:00
Jim Miller c4c4ff152f Add tgstorytime.com to allowed list for bulk_load param. 2015-04-15 19:52:19 -05:00
Jim Miller bc82d35c92 Update for PI forum thread. 2015-04-15 18:50:44 -05:00
36 changed files with 4930 additions and 3959 deletions
+6 -4
View File
@@ -10,8 +10,10 @@ FFDL, AKA fanficdownloader) project previously hosted as a
This program is available as a calibre plugin, a command-line
interface, and a web service.
*Currently, the project is the middle of being moved from code.google
to GitHub and being renamed. For now, continue to use
FanFictionDownLoader(FFDL).*
FanFicFare has now been launched. New versions, features and updates
will all be in FanFicFare.
[code.google project]: https://code.google.com/p/fanficdownloader/
There's additional info in the project [wiki] pages.
[code.google project]: http://google-opensource.blogspot.com/2015/03/farewell-to-google-code.html
[wiki]: https://github.com/JimmXinu/FanFicFare/wiki
+1 -1
View File
@@ -42,7 +42,7 @@ class FanFicFareBase(InterfaceActionBase):
description = _('UI plugin to download FanFiction stories from various sites.')
supported_platforms = ['windows', 'osx', 'linux']
author = 'Jim Miller'
version = (2, 2, 2)
version = (2, 2, 4)
minimum_calibre_version = (1, 48, 0)
#: This field defines the GUI plugin class that contains all the code
+32
View File
@@ -0,0 +1,32 @@
<hr />
<p>Plugin created by Jim Miller, borrowing heavily from Grant Drake's
'<a href="http://www.mobileread.com/forums/showthread.php?t=134856">Reading List</a>',
'<a href="http://www.mobileread.com/forums/showthread.php?t=126727">Extract ISBN</a>' and
'<a href="http://www.mobileread.com/forums/showthread.php?t=134000">Count Pages</a>'
plugins.</p>
<p>
Calibre officially distributes plugins from the mobileread.com forum site.
The official distro channel for this plugin is there: <a href="http://www.mobileread.com/forums/showthread.php?t=259221">FanFicFare</a>
</p>
<p> I also monitor the
<a href="http://groups.google.com/group/fanfic-downloader">general users
group</a> for the downloader. That covers the web application and CLI, too.
</p>
<p>
The source for this plugin is available at it's
<a href="https://github.com/JimmXinu/FanFicFare">project home</a>.
</p>
<hr />
<p>
See the <a href="https://github.com/JimmXinu/FanFicFare/wiki/Supportedsites">list of supported sites</a>.
</p>
<p>
Read the <a href="https://github.com/JimmXinu/FanFicFare/wiki/FAQs">FAQs</a>.
</p>
-28
View File
@@ -1,28 +0,0 @@
<hr />
<p>Plugin created by Jim Miller, borrowing heavily from Grant Drake's
'<a href="http://www.mobileread.com/forums/showthread.php?t=134856">Reading List</a>',
'<a href="http://www.mobileread.com/forums/showthread.php?t=126727">Extract ISBN</a>' and
'<a href="http://www.mobileread.com/forums/showthread.php?t=134000">Count Pages</a>'
plugins.</p>
<p>
Calibre officially distributes plugins from the mobileread.com forum site.
The official distro channel for this plugin is there: <a href="http://www.mobileread.com/forums/showthread.php?t=163261">FanFictionDownLoader</a>
</p>
<p> I also monitor the
<a href="http://groups.google.com/group/fanfic-downloader">general users
group</a> for the downloader. That covers the web application and CLI, too.
</p>
The source for this plugin is available at it's
<a href="https://github.com/JimmXinu/FanFicFare">project home</a>.
<hr />
<p>
See the <a href="https://github.com/JimmXinu/FanFicFare/wiki/Supportedsites">list of supported sites</a>.
</p>
<p>
Read the <a href="https://github.com/JimmXinu/FanFicFare/wiki/FAQs">FAQs</a>.
</p>
+185 -58
View File
@@ -17,12 +17,12 @@ try:
from PyQt5.Qt import (QDialog, QWidget, QVBoxLayout, QHBoxLayout, QGridLayout,
QLabel, QLineEdit, QFont, QWidget, QTextEdit, QComboBox,
QCheckBox, QPushButton, QTabWidget, QScrollArea,
QDialogButtonBox, QGroupBox, Qt )
QDialogButtonBox, QGroupBox, QButtonGroup, QRadioButton, Qt)
except ImportError as e:
from PyQt4.Qt import (QDialog, QWidget, QVBoxLayout, QHBoxLayout, QGridLayout,
QLabel, QLineEdit, QFont, QWidget, QTextEdit, QComboBox,
QCheckBox, QPushButton, QTabWidget, QScrollArea,
QDialogButtonBox, QGroupBox, Qt )
QDialogButtonBox, QGroupBox, QButtonGroup, QRadioButton, Qt)
try:
from calibre.gui2 import QVariant
del QVariant
@@ -39,9 +39,15 @@ else:
return [convert_qvariant(i) for i in x.toList()]
return x.toPyObject()
from calibre.gui2.ui import get_gui
from calibre.gui2 import dynamic, info_dialog, question_dialog
from calibre.constants import numeric_version as calibre_version
from calibre.gui2.ui import get_gui
from calibre.gui2.complete2 import EditWithComplete
try:
from calibre.ebooks.covers import generate_cover as cal_generate_cover
HAS_CALGC=True
except:
HAS_CALGC=False
# pulls in translation files for _() strings
try:
@@ -54,7 +60,6 @@ except NameError:
# out of the _() strings.
# I'm tempted to override _() to include them...
no_trans = { 'pini':'personal.ini',
'imgset':'\n\n[epub]\ninclude_images:true\nkeep_summary_html:true\nmake_firstimage_cover:true\n\n',
'gcset':'generate_cover_settings',
'ccset':'custom_columns_settings',
'gc':'Generate Cover',
@@ -70,7 +75,10 @@ no_trans = { 'pini':'personal.ini',
'p':'password',
}
from calibre_plugins.fanficfare_plugin.prefs import prefs, PREFS_NAMESPACE
from calibre_plugins.fanficfare_plugin.prefs \
import (prefs, PREFS_NAMESPACE, updatecalcover_order, calcover_save_options,
gencalcover_order, SAVE_YES, SAVE_NO)
from calibre_plugins.fanficfare_plugin.dialogs \
import (UPDATE, UPDATEALWAYS, collision_order, save_collisions, RejectListDialog,
EditTextDialog, IniTextDialog, RejectUrlEntry)
@@ -84,8 +92,6 @@ from calibre_plugins.fanficfare_plugin.common_utils \
from calibre_plugins.fanficfare_plugin.fff_util \
import (test_config)
from calibre.gui2.complete2 import EditWithComplete #MultiCompleteLineEdit
class RejectURLList:
def __init__(self,prefs):
self.prefs = prefs
@@ -205,10 +211,8 @@ class ConfigWidget(QWidget):
if 'Reading List' not in plugin_action.gui.iactions:
self.readinglist_tab.setEnabled(False)
self.generatecover_tab = GenerateCoverTab(self, plugin_action)
tab_widget.addTab(self.generatecover_tab, 'Generate Cover')
if 'Generate Cover' not in plugin_action.gui.iactions:
self.generatecover_tab.setEnabled(False)
self.calibrecover_tab = CalibreCoverTab(self, plugin_action)
tab_widget.addTab(self.calibrecover_tab, _('Calibre Cover'))
self.countpages_tab = CountPagesTab(self, plugin_action)
tab_widget.addTab(self.countpages_tab, 'Count Pages')
@@ -234,7 +238,6 @@ class ConfigWidget(QWidget):
prefs['fileform'] = unicode(self.basic_tab.fileform.currentText())
prefs['collision'] = save_collisions[unicode(self.basic_tab.collision.currentText())]
prefs['updatemeta'] = self.basic_tab.updatemeta.isChecked()
prefs['updatecover'] = self.basic_tab.updatecover.isChecked()
prefs['updateepubcover'] = self.basic_tab.updateepubcover.isChecked()
prefs['keeptags'] = self.basic_tab.keeptags.isChecked()
prefs['suppressauthorsort'] = self.basic_tab.suppressauthorsort.isChecked()
@@ -247,7 +250,6 @@ class ConfigWidget(QWidget):
prefs['updatedefault'] = self.basic_tab.updatedefault.isChecked()
prefs['deleteotherforms'] = self.basic_tab.deleteotherforms.isChecked()
prefs['adddialogstaysontop'] = self.basic_tab.adddialogstaysontop.isChecked()
prefs['includeimages'] = self.basic_tab.includeimages.isChecked()
prefs['lookforurlinhtml'] = self.basic_tab.lookforurlinhtml.isChecked()
prefs['checkforseriesurlid'] = self.basic_tab.checkforseriesurlid.isChecked()
prefs['checkforurlchange'] = self.basic_tab.checkforurlchange.isChecked()
@@ -273,17 +275,23 @@ class ConfigWidget(QWidget):
# if they've removed everything, reset to default.
prefs['personal.ini'] = get_resources('plugin-example.ini')
# Generate Covers tab
prefs['gcnewonly'] = self.generatecover_tab.gcnewonly.isChecked()
# Covers tab
prefs['updatecalcover'] = calcover_save_options[unicode(self.calibrecover_tab.updatecalcover.currentText())]
# for backward compatibility:
prefs['updatecover'] = prefs['updatecalcover'] == SAVE_YES
prefs['gencalcover'] = calcover_save_options[unicode(self.calibrecover_tab.gencalcover.currentText())]
prefs['calibre_gen_cover'] = self.calibrecover_tab.calibre_gen_cover.isChecked()
prefs['plugin_gen_cover'] = self.calibrecover_tab.plugin_gen_cover.isChecked()
prefs['gcnewonly'] = self.calibrecover_tab.gcnewonly.isChecked()
gc_site_settings = {}
for (site,combo) in self.generatecover_tab.gc_dropdowns.iteritems():
for (site,combo) in self.calibrecover_tab.gc_dropdowns.iteritems():
val = unicode(convert_qvariant(combo.itemData(combo.currentIndex())))
if val != 'none':
gc_site_settings[site] = val
#print("gc_site_settings[%s]:%s"%(site,gc_site_settings[site]))
prefs['gc_site_settings'] = gc_site_settings
prefs['allow_gc_from_ini'] = self.generatecover_tab.allow_gc_from_ini.isChecked()
prefs['gc_polish_cover'] = self.generatecover_tab.gc_polish_cover.isChecked()
prefs['allow_gc_from_ini'] = self.calibrecover_tab.allow_gc_from_ini.isChecked()
prefs['gc_polish_cover'] = self.calibrecover_tab.gc_polish_cover.isChecked()
# Count Pages tab
countpagesstats = []
@@ -396,21 +404,17 @@ class BasicTab(QWidget):
horz.addWidget(self.collision)
self.l.addLayout(horz)
horz = QHBoxLayout()
self.updatemeta = QCheckBox(_('Default Update Calibre &Metadata?'),self)
self.updatemeta.setToolTip(_("On each download, FanFicFare offers an option to update Calibre's metadata (title, author, URL, tags, custom columns, etc) from the web site. <br />This sets whether that will default to on or off. <br />Columns set to 'New Only' in the column tabs will only be set for new books."))
self.updatemeta.setChecked(prefs['updatemeta'])
self.l.addWidget(self.updatemeta)
horz.addWidget(self.updatemeta)
self.updateepubcover = QCheckBox(_('Default Update EPUB Cover when Updating EPUB?'),self)
self.updateepubcover.setToolTip(_("On each download, FanFicFare offers an option to update the book cover image <i>inside</i> the EPUB from the web site when the EPUB is updated.<br />This sets whether that will default to on or off."))
self.updateepubcover.setChecked(prefs['updateepubcover'])
self.l.addWidget(self.updateepubcover)
self.smarten_punctuation = QCheckBox(_('Smarten Punctuation (EPUB only)'),self)
self.smarten_punctuation.setToolTip(_("Run Smarten Punctuation from Calibre's Polish Book feature on each EPUB download and update."))
self.smarten_punctuation.setChecked(prefs['smarten_punctuation'])
if calibre_version >= (0, 9, 39):
self.l.addWidget(self.smarten_punctuation)
horz.addWidget(self.updateepubcover)
self.l.addLayout(horz)
cali_gb = groupbox = QGroupBox(_("Updating Calibre Options"))
self.l = QVBoxLayout()
@@ -421,11 +425,6 @@ class BasicTab(QWidget):
self.deleteotherforms.setChecked(prefs['deleteotherforms'])
self.l.addWidget(self.deleteotherforms)
self.updatecover = QCheckBox(_('Update Calibre Cover when Updating Metadata?'),self)
self.updatecover.setToolTip(_("Update calibre book cover image from EPUB when metadata is updated. (EPUB only.)\nDoesn't go looking for new images on 'Update Calibre Metadata Only'."))
self.updatecover.setChecked(prefs['updatecover'])
self.l.addWidget(self.updatecover)
self.keeptags = QCheckBox(_('Keep Existing Tags when Updating Metadata?'),self)
self.keeptags.setToolTip(_("Existing tags will be kept and any new tags added.\n%(cmplt)s and %(inprog)s tags will be still be updated, if known.\n%(lul)s tags will be updated if %(lus)s in %(is)s.\n(If Tags is set to 'New Only' in the Standard Columns tab, this has no effect.)")%no_trans)
self.keeptags.setChecked(prefs['keeptags'])
@@ -466,16 +465,16 @@ class BasicTab(QWidget):
self.showmarked.setChecked(prefs['showmarked'])
self.l.addWidget(self.showmarked)
self.smarten_punctuation = QCheckBox(_('Smarten Punctuation (EPUB only)'),self)
self.smarten_punctuation.setToolTip(_("Run Smarten Punctuation from Calibre's Polish Book feature on each EPUB download and update."))
self.smarten_punctuation.setChecked(prefs['smarten_punctuation'])
self.l.addWidget(self.smarten_punctuation)
self.autoconvert = QCheckBox(_("Automatically Convert new/update books?"),self)
self.autoconvert.setToolTip(_("Automatically call calibre's Convert for new/update books.\nConverts to the current output format as chosen in calibre's\nPreferences->Behavior settings."))
self.autoconvert.setChecked(prefs['autoconvert'])
self.l.addWidget(self.autoconvert)
self.show_est_time = QCheckBox(_("Show estimated time left?"),self)
self.show_est_time.setToolTip(_("When a Progress Bar is shown, show a rough estimate of the time left."))
self.show_est_time.setChecked(prefs['show_est_time'])
self.l.addWidget(self.show_est_time)
gui_gb = groupbox = QGroupBox(_("GUI Options"))
self.l = QVBoxLayout()
groupbox.setLayout(self.l)
@@ -495,16 +494,15 @@ class BasicTab(QWidget):
self.adddialogstaysontop.setChecked(prefs['adddialogstaysontop'])
self.l.addWidget(self.adddialogstaysontop)
self.show_est_time = QCheckBox(_("Show estimated time left?"),self)
self.show_est_time.setToolTip(_("When a Progress Bar is shown, show a rough estimate of the time left."))
self.show_est_time.setChecked(prefs['show_est_time'])
self.l.addWidget(self.show_est_time)
misc_gb = groupbox = QGroupBox(_("Misc Options"))
self.l = QVBoxLayout()
groupbox.setLayout(self.l)
# this is a cheat to make it easier for users to realize there's a new include_images features.
self.includeimages = QCheckBox(_("Include images in EPUBs?"),self)
self.includeimages.setToolTip(_("Download and include images in EPUB stories. This is equivalent to adding:%(imgset)s ...to the top of %(pini)s. Your settings in %(pini)s will override this.")%no_trans)
self.includeimages.setChecked(prefs['includeimages'])
self.l.addWidget(self.includeimages)
self.injectseries = QCheckBox(_("Inject calibre Series when none found?"),self)
self.injectseries.setToolTip(_("If no series is found, inject the calibre series (if there is one) so \nit appears on the FanFicFare title page(not cover)."))
self.injectseries.setChecked(prefs['injectseries'])
@@ -708,6 +706,7 @@ class ReadingListTab(QWidget):
self.send_lists_box.update_items_cache(reading_lists)
self.send_lists_box.setText(prefs['send_lists'])
horz.addWidget(self.send_lists_box)
self.send_lists_box.setCursorPosition(0)
self.l.addLayout(horz)
self.addtoreadlists = QCheckBox(_('Add new/updated stories to "To Read" Reading List(s).'),self)
@@ -724,6 +723,7 @@ class ReadingListTab(QWidget):
self.read_lists_box.update_items_cache(reading_lists)
self.read_lists_box.setText(prefs['read_lists'])
horz.addWidget(self.read_lists_box)
self.read_lists_box.setCursorPosition(0)
self.l.addLayout(horz)
self.addtolistsonread = QCheckBox(_('Add stories back to "Send to Device" Reading List(s) when marked "Read".'),self)
@@ -733,14 +733,19 @@ class ReadingListTab(QWidget):
self.l.insertStretch(-1)
class GenerateCoverTab(QWidget):
class CalibreCoverTab(QWidget):
def __init__(self, parent_dialog, plugin_action):
self.parent_dialog = parent_dialog
self.plugin_action = plugin_action
QWidget.__init__(self)
self.l = QVBoxLayout()
self.gencov_elements=[] ## used to disable/enable when gen
## cover is off/on. This is more
## about being a visual que than real
## necessary function.
topl = self.l = QVBoxLayout()
self.setLayout(self.l)
try:
@@ -748,8 +753,116 @@ class GenerateCoverTab(QWidget):
gc_settings = gc_plugin.get_saved_setting_names()
except KeyError:
gc_settings= []
label = QLabel(_('The %(gc)s plugin can create cover images for books using various metadata and configurations. If you have GC installed, FanFicFare can run GC on new downloads and metadata updates. Pick a GC setting by site or Default.')%no_trans)
label = QLabel(_("The Calibre cover image for a downloaded book can come"
" from the story site(if EPUB and images are enabled), or"
" from either Calibre's built-in random cover generator or"
" the %(gc)s plugin.")%no_trans)
label.setWordWrap(True)
self.l.addWidget(label)
self.l.addSpacing(5)
tooltip = _("Update Calibre book cover image from EPUB when Calibre metadata is updated.\n"
"Doesn't go looking for new images on 'Update Calibre Metadata Only'.\n"
"Cover in EPUB could be from site or previously injected into the EPUB.\n"
"This comes before Generate Cover so %(gc)s(Plugin) use the image if configured to.")%no_trans
horz = QHBoxLayout()
label = QLabel(_('Update Calibre Cover (from EPUB):'))
label.setToolTip(tooltip)
horz.addWidget(label)
self.updatecalcover = QComboBox(self)
for i in updatecalcover_order:
self.updatecalcover.addItem(i)
# back compat. If has own value, use.
if prefs['updatecalcover']:
self.updatecalcover.setCurrentIndex(self.updatecalcover.findText(calcover_save_options[prefs['updatecalcover']]))
elif prefs['updatecover']: # doesn't have own val, set YES if old value set.
self.updatecalcover.setCurrentIndex(self.updatecalcover.findText(calcover_save_options[SAVE_YES]))
else: # doesn't have own value, old value not set, NO.
self.updatecalcover.setCurrentIndex(self.updatecalcover.findText(calcover_save_options[SAVE_NO]))
self.updatecalcover.setToolTip(tooltip)
label.setBuddy(self.updatecalcover)
horz.addWidget(self.updatecalcover)
self.l.addLayout(horz)
tooltip = _("Generate a Calibre book cover image when Calibre metadata is updated.<br />"
"Defaults to 'Yes, Always' for backward compatibility and because %(gc)s(Plugin)"
" will only run if configured for Default or site.")%no_trans
horz = QHBoxLayout()
label = QLabel(_('Generate Calibre Cover:'))
label.setToolTip(tooltip)
horz.addWidget(label)
self.gencalcover = QComboBox(self)
for i in gencalcover_order:
self.gencalcover.addItem(i)
# back compat. If has own value, use.
# if prefs['gencalcover']:
self.gencalcover.setCurrentIndex(self.gencalcover.findText(calcover_save_options[prefs['gencalcover']]))
# elif prefs['gencover']: # doesn't have own val, set YES if old value set.
# self.gencalcover.setCurrentIndex(self.gencalcover.findText(calcover_save_options[SAVE_YES]))
# else: # doesn't have own value, old value not set, NO.
# self.gencalcover.setCurrentIndex(self.gencalcover.findText(calcover_save_options[SAVE_NO]))
self.gencalcover.setToolTip(tooltip)
label.setBuddy(self.gencalcover)
horz.addWidget(self.gencalcover)
self.l.addLayout(horz)
self.gencalcover.currentIndexChanged.connect(self.endisable_elements)
# can't be local or it's destroyed when __init__ is done and
# connected things don't fire.
self.gencov_rdgrp = QButtonGroup()
self.gencov_gb = QGroupBox()
horz = QHBoxLayout()
self.gencov_gb.setLayout(horz)
self.plugin_gen_cover = QRadioButton(_('Plugin %(gc)s')%no_trans,self)
self.plugin_gen_cover.setToolTip(_("Use plugin to create covers. Additional settings are below."))
self.gencov_rdgrp.addButton(self.plugin_gen_cover)
# always, new only, when no cover from site, inject yes/no...
self.plugin_gen_cover.setChecked(prefs['plugin_gen_cover'])
horz.addWidget(self.plugin_gen_cover)
self.gencov_elements.append(self.plugin_gen_cover)
self.calibre_gen_cover = QRadioButton(_('Calibre Generate Cover'),self)
self.calibre_gen_cover.setToolTip(_("Call Calibre's Edit Metadata Generate cover"
" feature to create a random cover each time"
" a story is downloaded or updated.<br />"
"Right click or long click the 'Generate cover'"
" button in Calibre's Edit Metadata to customize."))
self.gencov_rdgrp.addButton(self.calibre_gen_cover)
# always, new only, when no cover from site, inject yes/no...
self.calibre_gen_cover.setChecked(prefs['calibre_gen_cover'])
horz.addWidget(self.calibre_gen_cover)
self.gencov_elements.append(self.calibre_gen_cover)
#self.l.addLayout(horz)
self.l.addWidget(self.gencov_gb)
self.gcnewonly = QCheckBox(_("Generate Covers Only for New Books")%no_trans,self)
self.gcnewonly.setToolTip(_("Default is to generate a cover any time the calibre metadata is"
" updated.<br \>Used for both Calibre and Plugin generated covers."))
self.gcnewonly.setChecked(prefs['gcnewonly'])
self.l.addWidget(self.gcnewonly)
self.gencov_elements.append(self.gcnewonly)
self.gc_polish_cover = QCheckBox(_("Inject/update the cover inside EPUB"),self)
self.gc_polish_cover.setToolTip(_("Calibre's Polish feature will be used to inject or update the generated"
" cover into the EPUB ebook file.<br \>Used for both Calibre and Plugin generated covers."))
self.gc_polish_cover.setChecked(prefs['gc_polish_cover'])
self.l.addWidget(self.gc_polish_cover)
self.gencov_elements.append(self.gc_polish_cover)
self.gcp_gb = QGroupBox(_("%(gc)s(Plugin) Settings")%no_trans)
topl.addWidget(self.gcp_gb)
self.l = QVBoxLayout()
self.gcp_gb.setLayout(self.l)
self.gencov_elements.append(self.gcp_gb)
self.gencov_rdgrp.buttonClicked.connect(self.endisable_elements)
label = QLabel(_('The %(gc)s plugin can create cover images for books using various metadata (including existing cover image). If you have %(gc)s installed, FanFicFare can run %(gc)s on new downloads and metadata updates. Pick a %(gc)s setting by site and/or one to use by Default.')%no_trans)
label.setWordWrap(True)
self.l.addWidget(label)
self.l.addSpacing(5)
@@ -772,7 +885,7 @@ class GenerateCoverTab(QWidget):
horz = QHBoxLayout()
label = QLabel(site)
if site == _("Default"):
s = _("On Metadata update, run %(gc)s with this setting, if not selected for specific site.")%no_trans
s = _("On Metadata update, run %(gc)s with this setting, if there isn't a more specific setting below.")%no_trans
else:
no_trans['site']=site # not ideal, but, meh.
s = _("On Metadata update, run %(gc)s with this setting for %(site)s stories.")%no_trans
@@ -798,20 +911,34 @@ class GenerateCoverTab(QWidget):
self.sl.insertStretch(-1)
self.gcnewonly = QCheckBox(_("Run %(gc)s Only on New Books")%no_trans,self)
self.gcnewonly.setToolTip(_("Default is to run GC any time the calibre metadata is updated."))
self.gcnewonly.setChecked(prefs['gcnewonly'])
self.l.addWidget(self.gcnewonly)
self.allow_gc_from_ini = QCheckBox(_('Allow %(gcset)s from %(pini)s to override')%no_trans,self)
self.allow_gc_from_ini.setToolTip(_("The %(pini)s parameter %(gcset)s allows you to choose a GC setting based on metadata rather than site, but it's much more complex.<br \>%(gcset)s is ignored when this is off.")%no_trans)
self.allow_gc_from_ini.setToolTip(_("The %(pini)s parameter %(gcset)s allows you to choose a %(gc)s setting based on metadata"
" rather than site, but it's much more complex.<br \>%(gcset)s is ignored when this is off.")%no_trans)
self.allow_gc_from_ini.setChecked(prefs['allow_gc_from_ini'])
self.l.addWidget(self.allow_gc_from_ini)
self.gc_polish_cover = QCheckBox(_("Use calibre's Polish feature to inject/update the cover"),self)
self.gc_polish_cover.setToolTip(_("Calibre's Polish feature will be used to inject or update the generated cover into the ebook, EPUB only."))
self.gc_polish_cover.setChecked(prefs['gc_polish_cover'])
self.l.addWidget(self.gc_polish_cover)
# keep at end.
self.endisable_elements()
def endisable_elements(self,button=None):
"Clearing house function for setting elements of Calibre"
"Cover tab enabled/disabled depending on all factors."
## First, cover gen on/off
for e in self.gencov_elements:
e.setEnabled(calcover_save_options[unicode(self.gencalcover.currentText())] != SAVE_NO)
# next, disable plugin settings when using calibre gen cov.
if not self.plugin_gen_cover.isChecked():
self.gcp_gb.setEnabled(False)
# disable (but not enable) unsupported options.
if not HAS_CALGC:
self.calibre_gen_cover.setEnabled(False)
if not 'Generate Cover' in self.plugin_action.gui.iactions:
self.plugin_gen_cover.setEnabled(False)
self.gcp_gb.setEnabled(False)
class CountPagesTab(QWidget):
+4 -1
View File
@@ -122,7 +122,7 @@ gpstyle='QGroupBox {border:0; padding-top:10px; padding-bottom:0px; margin-botto
class RejectUrlEntry:
matchpat=re.compile(r"^(?P<url>[^,]+)(,(?P<fullnote>(((?P<title>.+) by (?P<auth>.+?)( - (?P<note>.+))?)|.*)))?$")
matchpat=re.compile(r"^(?P<url>[^,]+?)(,(?P<fullnote>(((?P<title>.+?) by (?P<auth>.+?)( - (?P<note>.+))?)|.*)))?$")
def __init__(self,url_or_line,note=None,title=None,auth=None,
addreasontext=None,fromline=False,book_id=None):
@@ -997,6 +997,7 @@ class RejectListTableWidget(QTableWidget):
note_cell.set_separator(None)
note_cell.setToolTip(_('Select or Edit Reject Note.'))
self.setCellWidget(row, 3, note_cell)
note_cell.setCursorPosition(0)
def remove_selected_rows(self):
self.setFocus()
@@ -1072,6 +1073,7 @@ class RejectListDialog(SizePersistedDialog):
label.setToolTip(_("This will be added to whatever note you've set for each URL above."))
horz.addWidget(label)
horz.addWidget(self.reason_edit)
self.reason_edit.setCursorPosition(0)
horz.insertStretch(-1)
layout.addLayout(horz)
@@ -1171,6 +1173,7 @@ class EditTextDialog(SizePersistedDialog):
self.l.addLayout(horz)
else:
self.l.addWidget(self.reason_edit)
self.reason_edit.setCursorPosition(0)
button_box = QDialogButtonBox(QDialogButtonBox.Ok | QDialogButtonBox.Cancel)
button_box.accepted.connect(self.accept)
+107 -76
View File
@@ -48,6 +48,13 @@ try:
except NameError:
pass # load_translations() added in calibre 1.9
try:
# should be present from cal2.3.0 on.
from calibre.ebooks.covers import generate_cover as cal_generate_cover
HAS_CALGC=True
except:
HAS_CALGC=False
from calibre_plugins.fanficfare_plugin.common_utils import (set_plugin_icon_resources, get_icon,
create_menu_action_unique, get_library_uuid)
@@ -57,7 +64,8 @@ from calibre_plugins.fanficfare_plugin.fanficfare.geturls import get_urls_from_p
from calibre_plugins.fanficfare_plugin.fff_util import (get_fff_adapter, get_fff_config, get_fff_personalini)
from calibre_plugins.fanficfare_plugin.config import (permitted_values, rejecturllist)
from calibre_plugins.fanficfare_plugin.prefs import prefs
from calibre_plugins.fanficfare_plugin.prefs import (prefs, SAVE_YES, SAVE_NO,
SAVE_YES_IF_IMG, SAVE_YES_UNLESS_IMG)
from calibre_plugins.fanficfare_plugin.dialogs import (
AddNewDialog, UpdateExistingDialog,
LoopProgressDialog, UserPassDialog, AboutDialog, CollectURLDialog, RejectListDialog, EmailPassDialog,
@@ -350,7 +358,7 @@ class FanFicFarePlugin(InterfaceAction):
# get_resources will return a dictionary mapping names to bytes. Names that
# are not found in the zip file will not be in the returned dictionary.
text = get_resources('about.txt')
text = get_resources('about.html')
AboutDialog(self.gui,self.qaction.icon(),self.version + text).exec_()
def create_menu_item_ex(self, parent_menu, menu_text, image=None, tooltip=None,
@@ -1609,15 +1617,6 @@ class FanFicFarePlugin(InterfaceAction):
if not oldmi.languages:
mi.languages=['en']
if options['fileform'] == 'epub' and prefs['updatecover']:
existingepub = db.format(book_id,'EPUB',index_is_id=True, as_file=True)
epubmi = get_metadata(existingepub,'EPUB')
if epubmi.cover_data[1] is not None:
try:
db.set_cover(book_id, epubmi.cover_data[1])
except:
logger.info("Failed to set_cover, skipping")
# implement 'newonly' flags here by setting to the current
# value again.
if not book['added']:
@@ -1757,73 +1756,105 @@ class FanFicFarePlugin(InterfaceAction):
db.commit()
if 'Generate Cover' in self.gui.iactions and (book['added'] or not prefs['gcnewonly']):
#logger.debug("Do Generate Cover added:%s gcnewonly:%s"%(book['added'],prefs['gcnewonly']))
# force a refresh if generating cover so complex composite
# custom columns are current and correct
db.refresh_ids([book_id])
gc_plugin = self.gui.iactions['Generate Cover']
setting_name = None
if prefs['allow_gc_from_ini']:
if not configuration: # might already have it from allow_custcol_from_ini
configuration = get_fff_config(book['url'],options['fileform'])
# template => regexp to match => GC Setting to use.
# generate_cover_settings:
# ${category} => Buffy:? the Vampire Slayer => Buffy
# for line in configuration.getConfig('generate_cover_settings').splitlines():
# if "=>" in line:
# (template,regexp,setting) = map( lambda x: x.strip(), line.split("=>") )
for (template,regexp,setting) in configuration.get_generate_cover_settings():
value = Template(template).safe_substitute(book['all_metadata']).encode('utf8')
# print("%s(%s) => %s => %s"%(template,value,regexp,setting))
if re.search(regexp,value):
setting_name = setting
break
# updating calibre cover from book.
if options['fileform'] == 'epub' and (
(prefs['updatecover'] and not prefs['updatecalcover']) ## backward compat
or prefs['updatecalcover'] == SAVE_YES ## yes, always
or (prefs['updatecalcover'] == SAVE_YES_IF_IMG ## yes, if image.
and book['all_metadata']['cover_image'] )): # in ('specific','first','default','old')
existingepub = db.format(book_id,'EPUB',index_is_id=True, as_file=True)
epubmi = get_metadata(existingepub,'EPUB')
if epubmi.cover_data[1] is not None:
try:
db.set_cover(book_id, epubmi.cover_data[1])
except:
logger.info("Failed to set_cover, skipping")
# First, should cover generation happen at all?
if (book['added'] or not prefs['gcnewonly']) and ( # skip if not new book and gcnewonly is True
prefs['gencalcover'] == SAVE_YES ## yes, always
or (prefs['gencalcover'] == SAVE_YES_UNLESS_IMG ## yes, unless image.
and book['all_metadata']['cover_image'] not in ('specific','first','default')) ):
cover_generated = False # flag for polish below.
# Yes, should do gencov. Which?
if prefs['calibre_gen_cover'] and HAS_CALGC:
# calibre's builtin, if available.
cdata = cal_generate_cover(mi)
db.set_cover(book_id, cdata)
cover_generated = True
elif prefs['plugin_gen_cover'] and 'Generate Cover' in self.gui.iactions:
# plugin, if available.
#logger.debug("Do Generate Cover added:%s gcnewonly:%s"%(book['added'],prefs['gcnewonly']))
# force a refresh if generating cover so complex composite
# custom columns are current and correct
db.refresh_ids([book_id])
gc_plugin = self.gui.iactions['Generate Cover']
setting_name = None
if prefs['allow_gc_from_ini']:
if not configuration: # might already have it from allow_custcol_from_ini
configuration = get_fff_config(book['url'],options['fileform'])
# template => regexp to match => GC Setting to use.
# generate_cover_settings:
# ${category} => Buffy:? the Vampire Slayer => Buffy
# for line in configuration.getConfig('generate_cover_settings').splitlines():
# if "=>" in line:
# (template,regexp,setting) = map( lambda x: x.strip(), line.split("=>") )
for (template,regexp,setting) in configuration.get_generate_cover_settings():
value = Template(template).safe_substitute(book['all_metadata']).encode('utf8')
print("%s(%s) => %s => %s"%(template,value,regexp,setting))
if re.search(regexp,value):
setting_name = setting
break
if setting_name:
logger.debug("Generate Cover Setting from generate_cover_settings(%s)"%setting_name)
if setting_name not in gc_plugin.get_saved_setting_names():
logger.info("GC Name %s not found, discarding! (check personal.ini for typos)"%setting_name)
setting_name = None
if not setting_name and book['all_metadata']['site'] in prefs['gc_site_settings']:
setting_name = prefs['gc_site_settings'][book['all_metadata']['site']]
logger.debug("Generate Cover Setting from site(%s)"%setting_name)
if not setting_name and 'Default' in prefs['gc_site_settings']:
setting_name = prefs['gc_site_settings']['Default']
logger.debug("Generate Cover Setting from Default(%s)"%setting_name)
if setting_name:
logger.debug("Generate Cover Setting from generate_cover_settings(%s)"%setting_name)
if setting_name not in gc_plugin.get_saved_setting_names():
logger.info("GC Name %s not found, discarding! (check personal.ini for typos)"%setting_name)
setting_name = None
if not setting_name and book['all_metadata']['site'] in prefs['gc_site_settings']:
setting_name = prefs['gc_site_settings'][book['all_metadata']['site']]
logger.debug("Generate Cover Setting from site(%s)"%setting_name)
if not setting_name and 'Default' in prefs['gc_site_settings']:
setting_name = prefs['gc_site_settings']['Default']
logger.debug("Generate Cover Setting from Default(%s)"%setting_name)
if setting_name:
logger.debug("Running Generate Cover with settings %s."%setting_name)
realmi = db.get_metadata(book_id, index_is_id=True)
gc_plugin.generate_cover_for_book(realmi,saved_setting_name=setting_name)
if prefs['gc_polish_cover'] and \
options['fileform'] == "epub":
# set cover inside epub from calibre's polish feature
from calibre.ebooks.oeb.polish.main import polish, ALL_OPTS
from calibre.utils.logging import Log
from collections import namedtuple
# Couldn't find a better way to get the cover path.
cover_path = os.path.join(db.library_path, db.path(book_id, index_is_id=True), 'cover.jpg')
data = {'cover':cover_path}
#print("cover_path:%s"%cover_path)
opts = ALL_OPTS.copy()
opts.update(data)
O = namedtuple('Options', ' '.join(ALL_OPTS.iterkeys()))
opts = O(**opts)
log = Log(level=Log.DEBUG)
outfile = db.format_abspath(book_id, formmapping[options['fileform']], index_is_id=True)
#print("polish cover outfile:%s"%outfile)
polish({outfile:outfile}, opts, log, logger.info)
logger.debug("Running Generate Cover with settings %s."%setting_name)
realmi = db.get_metadata(book_id, index_is_id=True)
gc_plugin.generate_cover_for_book(realmi,saved_setting_name=setting_name)
cover_generated = True
if cover_generated and prefs['gc_polish_cover'] and \
options['fileform'] == "epub":
# set cover inside epub from calibre's polish feature
from calibre.ebooks.oeb.polish.main import polish, ALL_OPTS
from calibre.utils.logging import Log
from collections import namedtuple
# Couldn't find a better way to get the cover path.
cover_path = os.path.join(db.library_path,
db.path(book_id, index_is_id=True),
'cover.jpg')
data = {'cover':cover_path}
#print("cover_path:%s"%cover_path)
opts = ALL_OPTS.copy()
opts.update(data)
O = namedtuple('Options', ' '.join(ALL_OPTS.iterkeys()))
opts = O(**opts)
log = Log(level=Log.DEBUG)
outfile = db.format_abspath(book_id,
formmapping[options['fileform']],
index_is_id=True)
#print("polish cover outfile:%s"%outfile)
polish({outfile:outfile}, opts, log, logger.info)
def get_clean_reading_lists(self,lists):
if lists == None or lists.strip() == "" :
+1 -9
View File
@@ -18,15 +18,7 @@ from calibre_plugins.fanficfare_plugin.fanficfare.configurable import Configurat
from calibre_plugins.fanficfare_plugin.prefs import prefs
def get_fff_personalini():
if prefs['includeimages']:
# this is a cheat to make it easier for users.
return '''[epub]
include_images:true
keep_summary_html:true
make_firstimage_cover:true
''' + prefs['personal.ini']
else:
return prefs['personal.ini']
return prefs['personal.ini']
def get_fff_config(url,fileform="epub",personalini=None):
if not personalini:
+4
View File
@@ -154,6 +154,7 @@ def do_download_for_worker(book,options,notification=lambda x,y:x):
if options['collision'] in (CALIBREONLY):
logger.info("Skipping CALIBREONLY 'update' down inside worker--this shouldn't be happening...")
book['comment'] = 'Metadata collected.'
book['all_metadata'] = story.getAllMetadata(removeallentities=True)
## checks were done earlier, it's new or not dup or newer--just write it.
elif options['collision'] in (ADDNEW, SKIP, OVERWRITE, OVERWRITEALWAYS) or \
@@ -178,6 +179,7 @@ def do_download_for_worker(book,options,notification=lambda x,y:x):
logger.info("write to %s"%outfile)
writer.writeStory(outfilename=outfile, forceOverwrite=True)
book['comment'] = 'Download %s completed, %s chapters.'%(options['fileform'],story.getMetadata("numChapters"))
book['all_metadata'] = story.getAllMetadata(removeallentities=True)
## checks were done earlier, just update it.
elif 'epub_for_update' in book and options['collision'] in (UPDATE, UPDATEALWAYS):
@@ -197,6 +199,7 @@ def do_download_for_worker(book,options,notification=lambda x,y:x):
if options['collision'] == UPDATE:
if chaptercount == urlchaptercount:
book['comment']=_("Already contains %d chapters. Reuse as is.")%chaptercount
book['all_metadata'] = story.getAllMetadata(removeallentities=True)
book['outfile'] = book['epub_for_update'] # for anthology merge ops.
return book
@@ -215,6 +218,7 @@ def do_download_for_worker(book,options,notification=lambda x,y:x):
book['comment'] = _('Update %s completed, added %s chapters for %s total.')%\
(options['fileform'],(urlchaptercount-chaptercount),urlchaptercount)
book['all_metadata'] = story.getAllMetadata(removeallentities=True)
if options['smarten_punctuation'] and options['fileform'] == "epub" \
and calibre_version >= (0, 9, 39):
+41 -9
View File
@@ -235,9 +235,18 @@ connect_timeout:60.0
## To use this, make sure you go to the Generate Cover tab in FanFicFare
## config and check 'Allow generate_cover_settings from personal.ini
## to override'
## You can check for an existing cover image in the metadata
## cover_image. cover_image can be:
## specific -- The site has an image that is specifically this story's cover
## first -- The first image in the story or story desc is used as cover
## default -- A default_cover_image was used
## old -- There was already a cover image in the epub.
## This is useful because GC plugin can use the existing image.
#generate_cover_settings:
# ${category} => Buffy:? [tT]he Vampire Slayer => BuffyCover
# ${category} => Star Trek => StarTrekCover
# ${cover_image} => => CompleteCover
# ${cover_image} => (specific|first|default) => CoverWithOrigImage
## If set false, the summary will have all html stripped.
## Both this and include_images must be true to get images in the
@@ -442,12 +451,12 @@ output_css:
## include images from img tags in the body and summary of
## stories. Images will be converted to jpg for size if possible.
## include_images is *only* available in epub and html output format.
#include_images:false
include_images:true
## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
#make_firstimage_cover: false
make_firstimage_cover: true
## If set, the epub will never have a cover, even include_images is on
## and the site has specific cover images.
@@ -745,7 +754,7 @@ extraships:Spike/Buffy
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -930,7 +939,7 @@ extraships:Harry Potter/Hermione Granger
#username:YourName
#password:yourpassword
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1070,6 +1079,23 @@ extracategories:In Death
## cover image. This lets you exclude them.
cover_exclusion_regexp:/public/style_emoticons/.*
[it-could-happen.net]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## RPF == http://en.wikipedia.org/wiki/Real_person_fiction
## Site dedicated to these categories/characters/ships
extracategories:Glee RPF
extracharacters:Darren Criss, Chris Colfer
extraships:Darren Criss/Chris Colfer
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
[ksarchive.com]
## Site dedicated to these categories/characters/ships
extracategories:Star Trek
@@ -1089,7 +1115,7 @@ eroticatags_label:Erotica Tags
extra_titlepage_entries: eroticatags
[lotrfanfiction.com]
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1343,7 +1369,7 @@ extracategories:Transgender
## confirm they are adult for adult content.
#is_adult:true
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1367,7 +1393,7 @@ extraships:Harry Potter/Draco Malfoy
## personal.ini, not defaults.ini.
#is_adult:true
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1381,7 +1407,7 @@ extracategories:Criminal Minds
## personal.ini, not defaults.ini.
#is_adult:true
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1454,6 +1480,8 @@ extra_titlepage_entries:readings,awards
awards_label:Awards
readings_label:Readings
## some sites include images that we don't ever want becoming the
## cover image. This lets you exclude them.
cover_exclusion_regexp:art/.*Awards.jpg
[voracity2.e-fic.com]
@@ -1650,6 +1678,10 @@ dateUpdated_format:%%Y-%%m-%%d %%H:%%M:%%S
## to on, but can be switched off if it is found to cause problems.
fix_fimf_blockquotes:true
## some sites include images that we don't ever want becoming the
## cover image. This lets you exclude them.
cover_exclusion_regexp:/images/emoticons/
## Site dedicated to these categories/characters/ships
extracategories:My Little Pony: Friendship is Magic
@@ -1733,7 +1765,7 @@ extraships:InuYasha/Kagome
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
+30 -4
View File
@@ -15,9 +15,32 @@ import copy
from calibre.utils.config import JSONConfig
from calibre.gui2.ui import get_gui
from calibre_plugins.fanficfare_plugin.dialogs import UPDATE
from calibre_plugins.fanficfare_plugin.dialogs import SAVE_UPDATE
from calibre_plugins.fanficfare_plugin.common_utils import get_library_uuid
# Show translated strings, but save the same string in prefs so your
# prefs are the same in different languages.
YES=_('Yes, Always')
SAVE_YES='Yes'
YES_IF_IMG=_('Yes, if EPUB has a cover image')
SAVE_YES_IF_IMG='Yes, if img'
YES_UNLESS_IMG=_('Yes, unless FanFicFare found a cover image')
SAVE_YES_UNLESS_IMG='Yes, unless img'
NO=_('No')
SAVE_NO='No'
calcover_save_options = {
YES:SAVE_YES,
SAVE_YES:YES,
YES_IF_IMG:SAVE_YES_IF_IMG,
SAVE_YES_IF_IMG:YES_IF_IMG,
YES_UNLESS_IMG:SAVE_YES_UNLESS_IMG,
SAVE_YES_UNLESS_IMG:YES_UNLESS_IMG,
NO:SAVE_NO,
SAVE_NO:NO,
}
updatecalcover_order=[YES,YES_IF_IMG,NO]
gencalcover_order=[YES,YES_UNLESS_IMG,NO]
# if don't have any settings for FanFicFarePlugin, copy from
# predecessor FanFictionDownLoaderPlugin.
FFDL_PREFS_NAMESPACE = 'FanFictionDownLoaderPlugin'
@@ -35,7 +58,6 @@ Dup from another site'''
default_prefs['reject_always'] = False
default_prefs['updatemeta'] = True
default_prefs['updatecover'] = False
default_prefs['updateepubcover'] = False
default_prefs['keeptags'] = False
default_prefs['suppressauthorsort'] = False
@@ -46,10 +68,9 @@ default_prefs['autoconvert'] = False
default_prefs['urlsfromclip'] = True
default_prefs['updatedefault'] = True
default_prefs['fileform'] = 'epub'
default_prefs['collision'] = UPDATE
default_prefs['collision'] = SAVE_UPDATE
default_prefs['deleteotherforms'] = False
default_prefs['adddialogstaysontop'] = False
default_prefs['includeimages'] = False
default_prefs['lookforurlinhtml'] = False
default_prefs['checkforseriesurlid'] = True
default_prefs['checkforurlchange'] = True
@@ -63,6 +84,11 @@ default_prefs['addtolists'] = False
default_prefs['addtoreadlists'] = False
default_prefs['addtolistsonread'] = False
default_prefs['updatecalcover'] = None
default_prefs['gencalcover'] = SAVE_YES
default_prefs['updatecover'] = False
default_prefs['calibre_gen_cover'] = False
default_prefs['plugin_gen_cover'] = True
default_prefs['gcnewonly'] = False
default_prefs['gc_site_settings'] = {}
default_prefs['allow_gc_from_ini'] = True
+6
View File
@@ -0,0 +1,6 @@
# Translations
If you're interested in helping provide translations for this project,
please use the
[Transifex](https://www.transifex.com/projects/p/calibre-plugins/resources/)
website to add translations to this, or other calibre plugins that support it.
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+7
View File
@@ -134,6 +134,7 @@ import adapter_fhsarchivecom
import adapter_fanfictionjunkiesde
import adapter_devianthearts
import adapter_tgstorytimecom
import adapter_itcouldhappennet
## This bit of complexity allows adapters to be added by just adding
## importing. It eliminates the long if/else clauses we used to need
@@ -196,6 +197,12 @@ def getAdapter(config,url,anyurl=False):
def getConfigSections():
return [cls.getConfigSection() for cls in __class_list]
def get_bulk_load_sites():
# for now, all eFiction Base adapters are assumed to allow bulk_load.
return [cls.getConfigSection().replace('www.','') for cls in
filter( lambda x : issubclass(x,base_efiction_adapter.BaseEfictionAdapter),
__class_list)]
def getSiteExamples():
l=[]
for cls in sorted(__class_list, key=lambda x : x.getConfigSection()):
+4 -1
View File
@@ -176,7 +176,10 @@ class FimFictionNetSiteAdapter(BaseSiteAdapter):
if coverurl.startswith('//'): # fix for img urls missing 'http:'
coverurl = "http:"+coverurl
if get_cover:
self.setCoverImage(self.url,coverurl)
# try setting from href, if fails, try using the img src
if self.setCoverImage(self.url,coverurl)[0] == "failedtoload":
coverurl = storyImage.find('img')['src']
self.setCoverImage(self.url,coverurl)
coverSource = storyImage.find('a', {'class':'source'})
if coverSource:
@@ -0,0 +1,46 @@
# -*- coding: utf-8 -*-
# Copyright 2015 FanFicFare team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Software: eFiction
import re
from base_efiction_adapter import BaseEfictionAdapter
class ItCouldHappenNetSiteAdapter(BaseEfictionAdapter):
@staticmethod
def getSiteDomain():
return 'it-could-happen.net'
@classmethod
def getSiteAbbrev(seluuf):
return 'ich'
@classmethod
def getDateFormat(self):
return "%B %d, %Y"
def handleMetadataPair(self, key, value):
# This site is all one 'category' as it's usually defined and
# uses Category for what is usually genre.
if key == 'Categories':
for val in re.split("\s*,\s*", value):
self.story.addToList('genre', val)
else:
super(ItCouldHappenNetSiteAdapter, self).handleMetadataPair(key, value)
def getClass():
return ItCouldHappenNetSiteAdapter
+129 -94
View File
@@ -21,7 +21,7 @@ logger = logging.getLogger(__name__)
import re
import urllib2
from .. import BeautifulSoup as bs
#from .. import BeautifulSoup as bs
from ..htmlcleanup import stripHTML
from .. import exceptions as exceptions
@@ -40,22 +40,22 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
self.password = ""
self.is_adult=False
# get storyId from url
self.story.setMetadata('storyId',self.parsedUrl.path.split('/',)[2].split(':')[0])
if 'storyInfo' in self.story.getMetadata('storyId'):
self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
# normalized story URL.
self._setURL('http://' + self.getSiteDomain() + '/s/'+self.story.getMetadata('storyId'))
# Each adapter needs to have a unique site abbreviation.
self.story.setMetadata('siteabbrev','strol')
# The date format will vary from site to site.
# http://docs.python.org/library/datetime.html#strftime-strptime-behavior
self.dateformat = "%Y-%m-%d"
@staticmethod # must be @staticmethod, don't remove it.
def getSiteDomain():
# The site domain. Does have www here, if it uses it.
@@ -78,7 +78,7 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
or "Access to unlinked chapters requires" in data:
self.needToLogin = True
return self.needToLogin
def performLogin(self, url):
params = {}
@@ -91,13 +91,13 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
params['rememberMe'] = '1'
params['page'] = 'http://'+self.getSiteDomain()+'/'
params['submit'] = 'Login'
loginUrl = 'http://' + self.getSiteDomain() + '/login.php'
logger.debug("Will now login to URL (%s) as (%s)" % (loginUrl,
params['theusername']))
d = self._fetchUrl(loginUrl, params,usecache=False)
if "My Account" not in d : #Member Account
logger.info("Failed to login to URL %s as %s" % (loginUrl,
params['theusername']))
@@ -137,22 +137,22 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
# need to log in for this one.
self.performLogin(url)
data = self._fetchUrl(url+":i",usecache=False)
if "Access denied. This story has not been validated by the adminstrators of this site." in data:
raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")
elif "Error! The story you're trying to access is being filtered by your choice of contents filtering." in data:
raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Error! The story you're trying to access is being filtered by your choice of contents filtering.")
# use BeautifulSoup HTML parser to make everything easier to find.
soup = bs.BeautifulSoup(data)
soup = self.make_soup(data)
#print data
# Now go hunting for all the meta data and the chapter list.
## Title
a = soup.find('h1')
self.story.setMetadata('title',stripHTML(a))
notice = soup.find('div', {'class' : 'notice'})
if notice:
self.story.setMetadata('notice',unicode(notice))
@@ -178,7 +178,7 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
page=0
i=0
while i == 0:
asoup = bs.BeautifulSoup(self._fetchUrl(self.story.getList('authorUrl')[0]+"/"+str(page)))
asoup = self.make_soup(self._fetchUrl(self.story.getList('authorUrl')[0]+"/"+str(page)))
a = asoup.findAll('td', {'class' : 'lc2'})
for lc2 in a:
@@ -196,8 +196,8 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
score = lc2.findNext('th', {'class' : 'num'}).text
if score != '-':
self.story.setMetadata('score', score)
self.story.setMetadata('score', score)
lc4 = lc2.findNext('td', {'class' : 'lc4'})
desc = lc4.contents[0]
@@ -212,7 +212,7 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
self.story.setMetadata('seriesUrl',seriesUrl)
series_name = stripHTML(a)
logger.debug("Series name= %s" % series_name)
series_soup = bs.BeautifulSoup(self._fetchUrl(seriesUrl))
series_soup = self.make_soup(self._fetchUrl(seriesUrl))
if series_soup:
logger.debug("Retrieving Series - looking for name")
series_name = series_soup.find('span', {'id' : 'ptitle'}).text.partition('')[0]
@@ -221,7 +221,7 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
desc = lc4.contents[2]
# Check if series is in a universe
universe_url = self.story.getList('authorUrl')[0] + "&type=uni"
universes_soup = bs.BeautifulSoup(self._fetchUrl(universe_url) )
universes_soup = self.make_soup(self._fetchUrl(universe_url) )
logger.debug("Universe url='{0}'".format(universe_url))
if universes_soup:
universes = universes_soup.findAll('div', {'class' : 'ser-box'})
@@ -255,7 +255,7 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
universe_name = stripHTML(a)
universeUrl = 'http://'+self.host+a['href']
logger.debug("Retrieving Universe - about to get page - universeUrl='{0}".format(universeUrl))
universe_soup = bs.BeautifulSoup(self._fetchUrl(universeUrl))
universe_soup = self.make_soup(self._fetchUrl(universeUrl))
logger.debug("Retrieving Universe - have page")
if universe_soup:
logger.debug("Retrieving Universe - looking for name")
@@ -274,7 +274,7 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
pass
self.setDescription('http://'+self.host+'/s/'+self.story.getMetadata('storyId'),desc)
for b in lc4.findAll('b'):
#logger.debug('Getting metadata: "%s"' % b)
label = b.text
@@ -284,24 +284,24 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
else:
value = b.nextSibling
#logger.debug('label: "%s", value: "%s"' % (label, value))
if 'Sex' in label:
self.story.setMetadata('rating', value)
if 'Tags' in label or 'Codes' in label:
for code in re.split(r'\s*,\s*', value.strip()):
self.story.addToList('sitetags',code)
if 'Posted' in label:
self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
if 'Concluded' in label:
self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
if 'Updated' in label:
self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
#
#
status = lc4.find('span', {'class' : 'ab'})
if status != None:
self.story.setMetadata('status', 'In-Progress')
@@ -312,111 +312,146 @@ class StoriesOnlineNetAdapter(BaseSiteAdapter):
else:
self.story.setMetadata('status', 'Completed')
# grab the text for an individual chapter.
def getChapterText(self, url):
logger.debug('Getting chapter text from: %s' % url)
soup = bs.BeautifulSoup(self._fetchUrl(url),
selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
div = soup.find('div', {'id' : 'story'})
soup = self.make_soup(self._fetchUrl(url))
chaptertag = soup.find('div', {'id' : 'story'})
if not chaptertag:
logger.debug("div id=story not found, try article")
chaptertag = soup.find('article', {'id' : 'story'})
# some big chapters are split over several pages
pager = div.find('span', {'class' : 'pager'})
pager = chaptertag.find('span', {'class' : 'pager'})
self.cleanPage(chaptertag)
if pager != None:
a = pager.previousSibling
while a != None:
logger.debug("before pager: {0}".format(a))
b = a.previousSibling
a.extract()
a = b
urls=pager.findAll('a')
urls=urls[:len(urls)-1]
logger.debug("pager urls:%s"%urls)
pager.extract()
div.contents = div.contents[2:]
# logger.debug(div)
chaptertag.contents = chaptertag.contents[2:]
for ur in urls:
soup = bs.BeautifulSoup(self._fetchUrl("http://"+self.getSiteDomain()+ur['href']),
selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
div1 = soup.find('div', {'id' : 'story'})
# Find the "Continues" marker on the current page and remove everything after that.
continues = div.find('span', {'class' : 'conTag'})
if continues != None:
while continues != None:
# logger.debug("removing end: {0}".format(continues))
b = continues.nextSibling
continues.extract()
continues = b
soup = self.make_soup(self._fetchUrl("http://"+self.getSiteDomain()+ur['href']))
# Find the "Continued" marker and delete everything before that
continued = div1.find('span', {'class' : 'conTag'})
if continued != None:
a = continued.previousSibling
while a != None:
# logger.debug("before conTag: {0}".format(a))
b = a.previousSibling
a.extract()
a = b
# Remove the pager from the end if this is the last page
endPager = div1.find('span', {'class' : 'pager'})
if endPager != None:
b = endPager.nextSibling
while endPager != None:
logger.debug("removing end: {0}".format(endPager))
b = endPager.nextSibling
endPager.extract()
endPager = b
div1.contents = div1.contents[:len(div1) - 2]
# logger.debug("after removing pager: {0}".format(div1))
for tag in div1.contents[2:]:
div.append(tag)
pagetag = soup.find('div', {'id' : 'story'})
if not pagetag:
logger.debug("div id=story not found, try article")
pagetag = soup.find('article', {'id' : 'story'})
# If it is a chapter, there are dates at the start for when it was posted or modified. These plus
# everything before them can be discarded.
postedDates = div.findAll('div', {'class' : 'date'})
self.cleanPage(pagetag)
for tag in pagetag.contents[2:]:
chaptertag.append(tag)
if None == chaptertag:
raise exceptions.FailedToDownload("Error downloading Chapter: %s! Missing required element!" % url)
return self.utf8FromSoup(url,chaptertag)
def cleanPage(self,pagetag):
"Consolidate 'page' clean up code so it can be called."
# some big chapters are split over several pages
# remove FIRST pager and everything before it.
tag = pagetag.find('span', {'class' : 'pager'})
while tag != None:
# logger.debug("remove before pager: {0}".format(tag))
prev = tag.previousSibling
tag.extract()
tag = prev
# Find the "Continues" marker on the current page and
# remove everything after that. This is actually
# effecting the *previous* 'page'. EXCEPT!--they are
# putting a 'conTag' at the *top* now, too. So this
# was nuking every page but the first and last. Now
# only if 'Continues'
for contag in pagetag.findAll('span', {'class' : 'conTag'}):
# remove everything after continues...
if 'Continuation' in contag.text:
tag = contag
while tag != None:
# logger.debug("remove before Continuation: {0}".format(tag))
prev = tag.previousSibling
tag.extract()
tag = prev
elif 'Continues' in contag.text:
tag = contag
while tag != None:
# logger.debug("remove after Continues: {0}".format(tag))
nxt = tag.nextSibling
tag.extract()
tag = nxt
# some big chapters are split over several pages
# remove LAST pager and everything before it.
# Only needed on last page.
tag = pagetag.find('span', {'class' : 'pager'})
while tag != None:
# logger.debug("remove after pager: {0}".format(tag))
nxt = tag.nextSibling
tag.extract()
tag = nxt
# If it is a chapter, there are dates at the start for when it was posted or modified. These plus
# everything before them can be discarded.
postedDates = pagetag.findAll('div', {'class' : 'date'})
if postedDates:
a = postedDates[0].previousSibling
while a != None:
# logger.debug("before dates: {0}".format(a))
# logger.debug("before dates: {0}".format(a))
b = a.previousSibling
a.extract()
a = b
for a in div.findAll('div', {'class' : 'date'}):
for a in pagetag.findAll('div', {'class' : 'date'}):
a.extract()
# For single chapter stories, there is a copyright statement. Remove this and everything
# before it.
copy = div.find('h4', {'class': 'copy'})
copy = pagetag.find('h4', {'class': 'copy'})
while copy != None:
# logger.debug("before copyright: {0}".format(copy))
# logger.debug("before copyright: {0}".format(copy))
b = copy.previousSibling
copy.extract()
copy = b
# For a story or the last chapter, remove voting form and the in library box
a = div.find('div', {'id' : 'vote-form'})
a = pagetag.find('div', {'id' : 'vote-form'})
if a != None:
a.extract()
a = div.find('div', {'id' : 'b-man-div'})
a = pagetag.find('div', {'id' : 'top-header'})
if a != None:
a.extract()
a = pagetag.find('div', {'id' : 'b-man-div'})
if a != None:
a.extract()
# Kill the "The End" header and everything after it.
a = div.find(['h2', 'h3'], {'class' : 'end'})
logger.debug("Chapter end= '{0}'".format(a))
# Kill the vote form and everything after it.
a = pagetag.find('div', {'class' : 'vform'})
# logger.debug("Chapter end= '{0}'".format(a))
while a != None:
b = a.nextSibling
a.extract()
a=b
# Kill the vote form and everything after it.
a = pagetag.find('h3', {'class' : 'end'})
# logger.debug("Chapter end= '{0}'".format(a))
while a != None:
b = a.nextSibling
a.extract()
a=b
foot = pagetag.find('footer')
if foot != None:
foot.extract()
if None == div:
raise exceptions.FailedToDownload("Error downloading Chapter: %s! Missing required element!" % url)
return self.utf8FromSoup(url,div)
+6 -2
View File
@@ -379,10 +379,12 @@ class BaseSiteAdapter(Configurable):
self.getConfig('allow_unsafe_filename')),
self._fetchUrlRaw,
cover=True)
self.story.setMetadata('cover_image','default')
# no new cover, set old cover, if there is one.
if not self.story.cover and self.oldcover:
self.story.oldcover = self.oldcover
self.story.setMetadata('cover_image','old')
# cheesy way to carry calibre bookmark file forward across update.
if self.calibrebookmark:
@@ -479,8 +481,10 @@ class BaseSiteAdapter(Configurable):
def setCoverImage(self,storyurl,imgurl):
if self.getConfig('include_images'):
self.story.addImgUrl(storyurl,imgurl,self._fetchUrlRaw,cover=True,
coverexclusion=self.getConfig('cover_exclusion_regexp'))
return self.story.addImgUrl(storyurl,imgurl,self._fetchUrlRaw,cover=True,
coverexclusion=self.getConfig('cover_exclusion_regexp'))
else:
return (None,None)
# bs3 & bs4 are different here.
# will move to a bs3 vs bs4 block if there's lots of changes.
@@ -89,7 +89,6 @@ class BaseEfictionAdapter(BaseSiteAdapter):
"""
return ["Windows-1252", "utf8"]
@classmethod
def getPathToArchive(cls):
"""
+14 -8
View File
@@ -144,14 +144,20 @@ def main(argv=None, parser=None, passed_defaultsini=None, passed_personalini=Non
else:
urls = args
for url in urls:
try:
do_download(url,
options,
passed_defaultsini,
passed_personalini)
except Exception, e:
print "URL(%s) Failed: Exception (%s). Run URL individually for more detail."%(url,e)
if len(urls) > 1:
for url in urls:
try:
do_download(url,
options,
passed_defaultsini,
passed_personalini)
except Exception, e:
print "URL(%s) Failed: Exception (%s). Run URL individually for more detail."%(url,e)
else:
do_download(urls[0],
options,
passed_defaultsini,
passed_personalini)
# make rest a function and loop on it.
def do_download(arg,
+6 -9
View File
@@ -118,15 +118,10 @@ def get_valid_set_options():
'universe_as_series':(['storiesonline.net'],None,boollist),
'strip_text_links':(['bloodshedverse.com'],None,boollist),
# eFiction Base
'bulk_load':(['fannation.shades-of-moonlight.com',
'fhsarchive.com',
'lotrfanfiction.com',
'themaplebookshelf.com',
'devianthearts.com',
'thehookupzone.net',
'libraryofmoria.com',
],None,boollist),
# eFiction Base adapters allow bulk_load
# kept forgetting to add them, so now it's automatic.
'bulk_load':(adapters.get_bulk_load_sites(),
None,boollist),
'include_logpage':(None,['epub'],boollist+['smart']),
@@ -136,6 +131,7 @@ def get_valid_set_options():
'grayscale_images':(None,['epub','html'],boollist),
'no_image_processing':(None,['epub','html'],boollist),
}
return dict(valdict)
def get_valid_scalar_entries():
@@ -163,6 +159,7 @@ def get_valid_scalar_entries():
'seriesHTML',
'langcode',
'output_css',
'cover_image',
])
def get_valid_entries():
+28 -7
View File
@@ -758,7 +758,7 @@ extraships:Spike/Buffy
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -925,7 +925,7 @@ extraships:Harry Potter/Hermione Granger
#username:YourName
#password:yourpassword
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1065,6 +1065,23 @@ extracategories:In Death
## cover image. This lets you exclude them.
cover_exclusion_regexp:/public/style_emoticons/.*
[it-could-happen.net]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content. In commandline version,
## this should go in your personal.ini, not defaults.ini.
#is_adult:true
## RPF == http://en.wikipedia.org/wiki/Real_person_fiction
## Site dedicated to these categories/characters/ships
extracategories:Glee RPF
extracharacters:Darren Criss, Chris Colfer
extraships:Darren Criss/Chris Colfer
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
[ksarchive.com]
## Site dedicated to these categories/characters/ships
extracategories:Star Trek
@@ -1084,7 +1101,7 @@ eroticatags_label:Erotica Tags
extra_titlepage_entries: eroticatags
[lotrfanfiction.com]
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1338,7 +1355,7 @@ extracategories:Transgender
## confirm they are adult for adult content.
#is_adult:true
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1362,7 +1379,7 @@ extraships:Harry Potter/Draco Malfoy
## personal.ini, not defaults.ini.
#is_adult:true
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1376,7 +1393,7 @@ extracategories:Criminal Minds
## personal.ini, not defaults.ini.
#is_adult:true
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
@@ -1639,6 +1656,10 @@ dateUpdated_format:%%Y-%%m-%%d %%H:%%M:%%S
## to on, but can be switched off if it is found to cause problems.
fix_fimf_blockquotes:true
## some sites include images that we don't ever want becoming the
## cover image. This lets you exclude them.
cover_exclusion_regexp:/images/emoticons/
## Site dedicated to these categories/characters/ships
extracategories:My Little Pony: Friendship is Magic
@@ -1722,7 +1743,7 @@ extraships:InuYasha/Kagome
## Site dedicated to these categories/characters/ships
extracategories:Lord of the Rings
## Virtually all eFiction-based sites allow downloading the whole story in
## Virtually all eFiction Base adapters allow downloading the whole story in
## bulk using the 'Print' feature. If 'bulk_load' is set to 'true', both
## metadata and chapters can be loaded in one step
bulk_load:true
+7
View File
@@ -24,6 +24,13 @@ class FailedToDownload(Exception):
def __str__(self):
return self.error
class RejectImage(Exception):
def __init__(self,error):
self.error=error
def __str__(self):
return self.error
class InvalidStoryURL(Exception):
def __init__(self,url,domain,example):
self.url=url
+21 -13
View File
@@ -44,6 +44,8 @@ imagetypes = {
try:
from calibre.utils.magick import Image
from StringIO import StringIO
from gif import GifInfo, CHECK_IS_ANIMATED
convtype = {'jpg':'JPG', 'png':'PNG'}
def convert_image(url,data,sizes,grayscale,
@@ -55,6 +57,10 @@ try:
owidth, oheight = img.size
nwidth, nheight = sizes
scaled, nwidth, nheight = fit_image(owidth, oheight, nwidth, nheight)
if normalize_format_name(img.format)=="gif" and GifInfo(StringIO(data),CHECK_IS_ANIMATED).frameCount > 1:
raise exceptions.RejectImage("Animated gifs come out purely--not going to use it.")
if scaled:
img.size = (nwidth, nheight)
export = True
@@ -224,7 +230,7 @@ langs = {
"Devanagari":"hi",
## These are from/for AO3:
u'العربية':'ar',
u'беларуская':'be',
u'Български език':'bg',
@@ -325,7 +331,7 @@ class InExMatch:
else:
retval = self.match == value
#print(">>>>>>>>>>>>>%s==%s r: %s,%s=%s"%(self.match,value,self.negate,retval, self.negate != retval))
return self.negate != retval
def __str__(self):
@@ -338,7 +344,7 @@ class InExMatch:
else:
s='='
return u'InExMatch(%s %s%s %s)'%(self.keys,f,s,self.match)
## metakey[,metakey]=~pattern
## metakey[,metakey]==string
## *for* part lines. Effect only when trailing conditional key=~regexp matches
@@ -358,7 +364,7 @@ def set_in_ex_clude(setting):
match = InExMatch(line)
dest.append([match,condmatch])
return dest
## Two or three part lines. Two part effect everything.
## Three part effect only those key(s) lists.
## pattern=>replacement
@@ -433,7 +439,7 @@ class Story(Configurable):
def join_list(self, key, vallist):
return self.getConfig("join_string_"+key,u", ").replace(SPACE_REPLACE,' ').join(map(unicode, vallist))
def setMetadata(self, key, value, condremoveentities=True):
# keep as list type, but set as only value.
@@ -445,20 +451,20 @@ class Story(Configurable):
self.metadata[key]=conditionalRemoveEntities(value)
else:
self.metadata[key]=value
if key == "language":
try:
# getMetadata not just self.metadata[] to do replace_metadata.
self.setMetadata('langcode',langs[self.getMetadata(key)])
except:
self.setMetadata('langcode','en')
if key == 'dateUpdated' and value:
# Last Update tags for Bill.
self.addToList('lastupdate',value.strftime("Last Update Year/Month: %Y/%m"))
self.addToList('lastupdate',value.strftime("Last Update: %Y/%m/%d"))
def do_in_ex_clude(self,which,value,key):
if value and which in self.in_ex_cludes:
include = 'include' in which
@@ -487,7 +493,7 @@ class Story(Configurable):
if include and keyfound and not found:
value = None
return value
def doReplacements(self,value,key,return_list=False,seen_list=[]):
value = self.do_in_ex_clude('include_metadata_pre',value,key)
@@ -526,7 +532,7 @@ class Story(Configurable):
# print("replacement,value:%s,%s->%s"%(replacement,value,regexp.sub(replacement,value)))
value = regexp.sub(replacement,value)
retlist = [value]
for val in retlist:
retlist = map(partial(self.do_in_ex_clude,'include_metadata_post',key=key),retlist)
retlist = map(partial(self.do_in_ex_clude,'exclude_metadata_post',key=key),retlist)
@@ -610,7 +616,7 @@ class Story(Configurable):
self.getMetadata('author', removeallentities, doreplacements)))
self.extendList("extratags",self.getConfigList("extratags"))
if self.getMetadataRaw('seriesUrl'):
self.setMetadata('seriesHTML',linkhtml%('series',self.getMetadata('seriesUrl', removeallentities, doreplacements),
self.getMetadata('series', removeallentities, doreplacements)))
@@ -703,10 +709,10 @@ class Story(Configurable):
for val in retlist:
newretlist.extend(self.doReplacements(val,listname,return_list=True))
retlist = newretlist
if removeallentities:
retlist = map(removeAllEntities,retlist)
retlist = filter( lambda x : x!=None and x!='' ,retlist)
# reorder ships so b/a and c/b/a become a/b and a/b/c. Only on '/',
@@ -874,6 +880,7 @@ class Story(Configurable):
self.imgurls.insert(0,imgurl)
newsrc = "images/cover.%s"%ext
self.cover=newsrc
self.setMetadata('cover_image','specific')
self.imgtuples.insert(0,{'newsrc':newsrc,'mime':mime,'data':data})
else:
self.imgurls.append(imgurl)
@@ -888,6 +895,7 @@ class Story(Configurable):
not (coverexclusion and re.search(coverexclusion,imgurl)):
newsrc = "images/cover.%s"%ext
self.cover=newsrc
self.setMetadata('cover_image','first')
self.imgtuples.append({'newsrc':newsrc,'mime':mime,'data':data})
self.imgurls.append(imgurl)
+283
View File
@@ -0,0 +1,283 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
A pure Python GIF metadata extractor.
Supports adjustable detail to fine-tune performance.
Example code and full epydoc docstrings included.
Uses:
- Identifying whether a GIF is static or animated.
- Extracting the dimensions, pixel aspect ratio, number of frames, loop count,
global palette or palette size, and background color.
- Extracting comments and other plaintext.
- Testing for various structural errors.
TODO:
- Provide basic support for XMP Metadata extraction
- http://en.wikipedia.org/wiki/Extensible_Metadata_Platform#Location_in_file_types
- http://www.matthewflickinger.com/lab/whatsinagif/bits_and_bytes.asp
- Generate test GIF with http://code.google.com/p/python-xmp-toolkit/
Changelog:
- 0.2.2: Audited the code and made some corrections.
- 0.2.1: 40% speed improvement (went from 15 to 9 seconds for 1000 images)
- 0.2.0: Feature-complete
- 0.1.0: Initial release
"""
__appname__ = "gif.py"
__author__ = "Stephan Sokolow (deitarion/SSokolow)"
__version__ = "0.2.2"
__license__ = "PSF License 2.4 or higher (The Python License)"
#{ Check Types (enum, numerical ordering is significant)
CHECK_IS_GIF_FILE = 0 #: Just check for a valid GIF header.
CHECK_IS_ANIMATED = 1 #: Check whether the file has more than one frame.
CHECK_COUNT_FRAMES = 2 #: Count the number of frames in the file.
CHECK_PARSE_PALETTE = 3 #: Parse the palette and resolve the background color.
CHECK_READ_COMMENTS = 4 #: Load comments (can sometimes be large) into the L{GifInfo} object.
CHECK_READ_ALL_TEXT = 5 #: Also load the contents of Plain Text extension blocks.
CHECK_ALL = CHECK_READ_ALL_TEXT #: alias to allow for future modifications
#{ Warning Codes (bitfield)
WARN_NONE = 0 #: No warnings
WARN_BAD_IMG = 1 #: Corruption (of the [sub]block size field(s)) or truncation detected in an image block
WARN_BAD_EXT = 2 #: Corruption (of the [sub]block size field(s)) or truncation detected in an extension block
WARN_BAD_SIZE = 4 #: An image block specifies dimensions exceeding the global canvas size
WARN_BAD_BGCOLOR = 8 #: The background color index specified is greater than the palette size
WARN_EOF = 16 #: File is missing it's trailer. (Corrupt elsewhere, truncated, or breaking spec by using EOF as the terminator.)
WARN_TRUNC = 32 #: File is definitively either truncated or corrupt. (An EOF was encountered part-way through a structure.)
WARN_LOOP_POS = 64 #: Netscape Application Extension block (animation-control) was present but not first in the file.
#}
import struct
#{ Structures used by GifInfo
gifHeaderStruct = struct.Struct('<xxxxxxHHBBB') #: File header
gifImageStruct = struct.Struct('<HHHHB') #: Image block header
gifExtenStruct = struct.Struct('<BB') #: Top-level extension block header
gifNetscapeStruct = struct.Struct('<BBH') #: NETSCAPE Loop-control sub-block
gifColorTripleStruct = struct.Struct('<BBB') #: RGB palette element
gifPlaintextStruct = struct.Struct('<BHHHHBBBB') #: Plain Text block header
#}
class BadHeaderException(Exception):
"""Raised when no valid GIF header is found"""
class GifInfo(object):
"""A class for loading and storing metadata from GIF files.
Accepts paths and file-like objects.
When using L{CHECK_ALL}, this can also be used to walk past a valid
GIF file in an un-delimited byte stream in order to identify the point at
which the following file starts. (It doesn't C{fh.seek(0)} or C{fh.close()} after
use)
"""
warnFlags = WARN_NONE #: A bit field of C{WARN_*} flags set by L{__init__}
checkLevel = CHECK_ALL #: Default C{CHECK_*} level used by L{__init__}
#{ Pre-defined "unset" values for GIF Metadata
path = None #: The path to the file, if one was passed to L{__init__}
version = None #: C{87a} or C{89a}
width = None
height = None
loopCount = None
pixelAspect = None
paletteSize = None #: Always calculated if a global palette is present
palette = None #: The palette as a list of integer RGB tuples. Requires L{CHECK_PARSE_PALETTE}.
bgColor = None #: Global background color as an RGB tuple. Requires L{CHECK_PARSE_PALETTE}.
comments = None #: Text in Comment (0xFE) extension blocks as a list of strings. Requires L{CHECK_READ_COMMENTS}
otherText = None #: Text in "Plain Text" (0x01) extension blocks as a list of strings. Requires L{CHECK_READ_ALL_TEXT}
frameCount = 0
#}
def __init__(self, fh, checkLevel=checkLevel):
"""
@param fh: A path or file-like object for a GIF file.
@param checkLevel: A C{CHECK_*} constant.
@raises BadHeaderException: The given file lacks a valid GIF header.
@raises IOError: The underlying C{open()} system call failed.
"""
self.checkLevel = checkLevel
if isinstance(fh, basestring):
self.path = fh
fh = open(fh, 'rb')
header = fh.read(gifHeaderStruct.size)
if len(header) < gifHeaderStruct.size:
raise BadHeaderException("File is too small to be a GIF")
self.version = header[3:6]
if header[0:3] != 'GIF' or self.version not in ['87a', '89a']:
raise BadHeaderException("File does not have a recognizable GIF header")
elif self.checkLevel <= CHECK_IS_GIF_FILE:
return
self.width, self.height, GCTF_Byte, bgColor, self.pixelAspect = gifHeaderStruct.unpack(header)
if self.pixelAspect:
self.pixelAspect = (self.pixelAspect + 15) / 64.0
rawPalette = self._getPalette(fh, GCTF_Byte)
self.paletteSize = int(len(rawPalette) / 3)
if self.checkLevel >= CHECK_PARSE_PALETTE:
self.palette = []
for pos in range(0, self.paletteSize):
self.palette.append(gifColorTripleStruct.unpack_from(rawPalette, pos * 3))
if self.paletteSize and bgColor > self.paletteSize:
self.warnFlags = self.warnFlags | WARN_BAD_BGCOLOR
elif self.palette:
self.bgColor = self.palette[bgColor]
# Iterate blocks
self.firstBlock = True
blocktype = self._read(fh, 1)
while not blocktype == chr(0x3B) and not self.warnFlags & WARN_EOF:
self._blockHandlers.get(blocktype, lambda x, y:'')(self, fh)
if self.checkLevel <= CHECK_IS_ANIMATED and self.frameCount > 1:
return
self.firstBlock = False
blocktype = self._read(fh, 1)
del self.firstBlock
def _handleImageBlock(self, fh):
""""""
self.frameCount += 1
try:
x, y, w, h, LCTF_Byte = gifImageStruct.unpack(self._read(fh, gifImageStruct.size))
except:
self.warnFlags = self.warnFlags | WARN_EOF | WARN_TRUNC
return
if x + w > self.width or y + h > self.height:
self.warnFlags = self.warnFlags | WARN_BAD_SIZE
self._getPalette(fh, LCTF_Byte) # Skip the local color table if present
fh.read(1) # Skip the LZW minimum code size.
# Skip content and test for the block terminator
if not self._skipSubBlocks(fh): # For example, if it's a zero-length string like EOF would return.
self.warnFlags = self.warnFlags | WARN_BAD_IMG
def _handleGenericExtensionBlock(self, fh):
"""@todo: Rewrite this so extension block types have method handlers."""
try:
extType, blkSize = gifExtenStruct.unpack(self._read(fh, gifExtenStruct.size))
except:
self.warnFlags = self.warnFlags | WARN_EOF | WARN_TRUNC
return
startOffset = fh.tell()
if extType == 0x01 and self.checkLevel >= CHECK_READ_ALL_TEXT: # Plain Text Block
self._read(fh, gifPlaintextStruct.size)
self.otherText = self.otherText or []
blkSize = self._read(fh, 1)
while blkSize and blkSize != '\x00':
self.otherText.append(self._read(fh, ord(blkSize)))
blkSize = self._read(fh, 1)
elif extType == 0xFE and self.checkLevel >= CHECK_READ_COMMENTS: # Comment Block
self.comments = self.comments or []
blkSize = self._read(fh, 1)
while blkSize and blkSize != '\x00':
self.comments.append(self._read(fh, ord(blkSize)))
blkSize = self._read(fh, 1)
elif extType == 0xFF: # Application Block
if blkSize == 0x0B and self._read(fh, blkSize) == "NETSCAPE2.0":
try:
a, b, self.loopCount = gifNetscapeStruct.unpack(self._read(fh, gifNetscapeStruct.size))
except:
self.warnFlags = self.warnFlags | WARN_EOF | WARN_TRUNC
return
if a != 3 and b != 1:
self.warnFlags = self.warnFlags | WARN_BAD_EXT
if not self.firstBlock:
self.warnFlags = self.warnFlags | WARN_LOOP_POS
else:
fh.seek( startOffset + blkSize ) # Skip the contents
# Test for the block terminator
if not self._skipSubBlocks(fh):
self.warnFlags = self.warnFlags | WARN_BAD_EXT
def _getPalette(self, handle, bitfield):
"""Using the size value from C{bitfield},
load the palette at C{handle}'s current file pointer position."""
if bitfield & int("10000000", 2):
nBits = bitfield & int("00000111", 2)
tableSize = 3 * 2**( nBits + 1 )
return handle.read(tableSize)
else:
return ''
def _read(self, handle, size):
"""Attempt to read the specified number of bytes. Set L{WARN_EOF} if
fewer are received."""
content = handle.read(size)
if len(content) < size:
self.warnFlags = self.warnFlags | WARN_EOF
return content
def _skipSubBlocks(self, handle):
"""Skip sub-blocks beginning at the current file pointer position
using fseek."""
offset = handle.tell()
blkSize = handle.read(1)
while blkSize and blkSize != '\x00':
offset += ord(blkSize) + 1
handle.seek(offset)
blkSize = handle.read(1)
return blkSize
_blockHandlers = {
chr(0x2C) : _handleImageBlock,
chr(0x21) : _handleGenericExtensionBlock,
}
def gif_is_animated(path):
"""A simple convenience function for testing whether a GIF is animated.
@rtype: C{bool}
"""
return GifInfo(file(path,'rb'), CHECK_IS_ANIMATED).frameCount > 1
if __name__ == '__main__':
from optparse import OptionParser
parser = OptionParser(description=__doc__.split('\n\n')[0],
version="%%prog v%s" % __version__, usage="%prog <path> ...")
opts, args = parser.parse_args()
if args:
for fpath in args:
try:
info = GifInfo(fpath, CHECK_COUNT_FRAMES)
warnFlags = (
(info.warnFlags & WARN_BAD_IMG and 'I' or ' ') +
(info.warnFlags & WARN_BAD_EXT and 'X' or ' ') +
(info.warnFlags & WARN_BAD_SIZE and 'C' or ' ') +
(info.warnFlags & WARN_BAD_BGCOLOR and 'B' or ' ') +
(info.warnFlags & WARN_EOF and 'E' or ' ') +
(info.warnFlags & WARN_TRUNC and 'T' or ' ') +
(info.warnFlags & WARN_LOOP_POS and 'L' or ' ')
)
print "[%s](%3s Frames): %s" % (warnFlags, info.frameCount, info.path)
except BadHeaderException, err:
print "%s: %s" % (str(err), fpath)
print "\nWarning Flags:"
print " I = Image Chunk Corruption/Truncation"
print " X = Extension Chunk Corruption/Truncation"
print " C = Image Chunk Dimensions Exceed Global Canvas"
print " B = Bad Background Color (Index Exceeds Palette Size)"
print " E = Unexpected EOF Encountered (Missing Image Terminator)"
print " T = EOF Encountered Within A Block Header (Corrupt or Truncated File)"
print " L = Loop-control block misplaced within the file"
print
print "Note: A nearly-threefold speed-up can be had by using CHECK_IS_ANIMATED rather than CHECK_COUNT_FRAMES"
+2 -2
View File
@@ -25,7 +25,7 @@ if __name__=="__main__":
exclude=['*.pyc','*~','*.xcf','*[0-9].png','*.po','*.pot','*default.mo','*Thumbs.db']
os.chdir('calibre-plugin')
files=['plugin-defaults.ini','plugin-example.ini','about.txt',
files=['plugin-defaults.ini','plugin-example.ini','about.html',
'images','translations']
files.extend(glob('*.py'))
files.extend(glob('plugin-import-name-*.txt'))
@@ -36,7 +36,7 @@ if __name__=="__main__":
os.chdir('../included_dependencies')
# 'a' for append
files=['six.py','bs4','html5lib','chardet']
files=['gif.py','six.py','bs4','html5lib','chardet']
createZipFile("../"+filename,"a",
files,
exclude=exclude)
+1 -1
View File
@@ -25,7 +25,7 @@ setup(
# Versions should comply with PEP440. For a discussion on single-sourcing
# the version across setup.py and the project code, see
# https://packaging.python.org/en/latest/single_source_version.html
version="2.2.2",
version="2.2.4",
description='A tool for downloading fanfiction to eBook formats',
long_description=long_description,
+1 -1
View File
@@ -1,6 +1,6 @@
# ffd-retief-hrd fanfictiondownloader fanficfare
application: fanficfare
version: 2-2-2
version: 2-2-4
runtime: python27
api_version: 1
threadsafe: true
+4 -16
View File
@@ -46,19 +46,7 @@
</p>
<h3>Changes:</h3>
<ul>
<li>Change the name to FanFicFare.</li>
<li>Fixes for literotica changes.</li>
<li>Fix error in debug output that uses wrong variable.</li>
<li>Change date format for onedirectionfanfiction.com.</li>
<li>Explicit html parse for BS4 when reading in existing epub. For cal1.48.</li>
<li>Fix for PI - eFiction Base adapter Anthology Series string.</li>
<li>Update translations.</li>
<li>Change some menu & gui details to be clearer.</li>
<li>Fix for ffnet's latest change to story/cover image.</li>
<li>Fix problems found during trawl through all sites -- thehookupzone.net and libraryofmoria.com now use eFiction Base.</li>
<li>Remove defunct sites - netraptor.org www.simplyundeniable.com www.wizardtales.net www.checkmated.com stargate-atlantis.org www.henneth-annun.net www.twiwrite.net www.dotmoon.net</li>
<li>Add option to check ffnet auth img vs cover img and not use author images as book covers. Defaults to on, add <b>skip_author_cover:false</b> to <b>[fanfiction.net]</b> section if you like author image covers.</li>
<li>Add site tgstorytime.com as an eFiction Base Adapter.</li>
<li>More fixes for storiesonline.net site changes.</li>
</ul>
<p>
Questions? Check out our
@@ -68,8 +56,8 @@
If you have any problems with this application, please
report them in
the <a href="http://groups.google.com/group/fanfic-downloader">FanFicFare Google Group</a>. The
<a href="http://fanfictiondownloader.appspot.com">FanFictionDownLoader(FFDL)
Version</a> is also available for you to use if necessary.
<a href="http://2-2-3.fanficfare.appspot.com">previous version
</a> is also available for you to use if necessary.
</p>
<div id='error'>
{{ error_message }}
@@ -130,7 +118,7 @@
Once you have calibre installed and running, inside
calibre, you can go to 'Get plugins to enhance calibre' or
'Get new plugins' and
install <a href="http://www.mobileread.com/forums/showthread.php?t=163261">FanFicFare</a>.
install <a href="http://www.mobileread.com/forums/showthread.php?t=259221">FanFicFare</a>.
</p>
</div>