Plugin only, optionally search epub text for story URL, option to GC only new.

Added tag FanFictionDownLoader-4.4.9 for changeset 5b585e031b72
Added tag calibre-plugin-1.5.20 for changeset 5b585e031b72
2026-09-15 11:36:30 +08:00 · 2012-05-18 12:11:16 -05:00 · 2012-05-12 18:49:02 -05:00 · 2012-05-12 18:48:40 -05:00 · 2012-05-12 18:47:22 -05:00 · 2012-05-12 18:44:40 -04:00
59 changed files with 4854 additions and 1027 deletions
@@ -0,0 +1,78 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
+<html>
+  <head>
+    <link href="/css/index.css" rel="stylesheet" type="text/css">
+    <title>FanFictionDownLoader (fanfiction.net, fanficauthors, fictionalley, ficwad to epub and HTML)</title>
+    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+    <script type="text/javascript">
+
+      var _gaq = _gaq || [];
+      _gaq.push(['_setAccount', 'UA-12136939-1']);
+      _gaq.push(['_trackPageview']);
+
+      (function() {
+      var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+      ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+      var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+      })();
+
+    </script>
+  </head>
+  <body>
+    <div id='main'>
+      <h1>
+	<a href="/" style="text-decoration: none; color: black;">FanFictionDownLoader</a>
+      </h1>
+      
+      <script type="text/javascript"><!--
+                                        google_ad_client = "ca-pub-0320924304307555";
+                                        /* Standard */
+                                        google_ad_slot = "8974025478";
+                                        google_ad_width = 468;
+                                        google_ad_height = 60;
+                                        //-->
+      </script>
+      <script type="text/javascript"
+              src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
+      </script>
+      <!--			<div id='yourfile'> -->
+      {{yourfile}}
+      <!--			</div> -->
+      
+      <div id='helpbox'>
+	{% for fic in fics %}
+	<p>
+  	  <a href="{{ fic.url }}" title="Link to original story"><span class="recent"><i>{{ fic.title }}</i></span></a>
+	      by <a href="{{ fic.authorUrl }}">{{ fic.author }}</a>  <b>Download Count:</b> {{ fic.count }} <br />
+	  <b>Word Count:</b> {{ fic.numWords }} <b>Chapter Count:</b> {{ fic.numChapters }}<br />
+	  {% if fic.category %} <b>Categories:</b> {{ fic.category }} <br /> {% endif %}
+	  {% if fic.genre %} <b>Genres:</b> {{ fic.genre }} <br /> {% endif %}
+	  {% if fic.language %} <b>Language:</b> {{ fic.language }} <br /> {% endif %}
+	  {% if fic.series %} <b>Series:</b> {{ fic.series }} <br /> {% endif %}
+	  {% if fic.characters %} <b>Characters:</b> {{ fic.characters }} <br /> {% endif %}
+	  {% if fic.status %} <b>Status:</b> {{ fic.status }} <br /> {% endif %}
+	  {% if fic.datePublished %} <b>Published:</b> {{ fic.datePublished }} <br /> {% endif %}
+	  {% if fic.dateUpdated %} <b>Last Updated:</b> {{ fic.dateUpdated }} <br /> {% endif %}
+	  {% if fic.dateCreated %} <b>Last Downloaded:</b> {{ fic.dateCreated }} <br /> {% endif %}
+	  {% if fic.rating %} <b>Rating:</b> {{ fic.rating }} <br /> {% endif %}
+	  {% if fic.warnings %} <b>Warnings:</b> {{ fic.warnings }} <br /> {% endif %}
+	  {% if fic.description %} <b>Summary:</b> {{ fic.description }} <br /> {% endif %}
+	</p>
+	{% endfor %}
+      </div>
+      
+      <script type="text/javascript"><!--
+                                        google_ad_client = "ca-pub-0320924304307555";
+                                        /* Standard */
+                                        google_ad_slot = "8974025478";
+                                        google_ad_width = 468;
+                                        google_ad_height = 60;
+                                        //-->
+      </script>
+      <script type="text/javascript"
+              src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
+      </script>				
+      
+    </div>
+  </body>
+</html>
@@ -1,6 +1,6 @@
 # ffd-retief-hrd fanfictiondownloader
 application: fanfictiondownloader
-version: 4-3-2
+version: 4-4-9
 runtime: python27
 api_version: 1
 threadsafe: true
@@ -35,8 +35,8 @@ handlers:
 - url: /.*
  script: main.app

-builtins:
- datastore_admin: on
+#builtins:
+#- datastore_admin: on

 libraries:
 - name: django
@@ -27,7 +27,7 @@ class FanFictionDownLoaderBase(InterfaceActionBase):
    description         = 'UI plugin to download FanFiction stories from various sites.'
    supported_platforms = ['windows', 'osx', 'linux']
    author              = 'Jim Miller'
-    version             = (1, 4, 4)
+    version             = (1, 5, 21)
    minimum_calibre_version = (0, 8, 30)

    #: This field defines the GUI plugin class that contains all the code
@@ -1,6 +1,6 @@
 <hr />

-<p>Created by Jim Miller, borrowing heavily from Grant Drake's
+<p>Plugin created by Jim Miller, borrowing heavily from Grant Drake's
 '<a href="http://www.mobileread.com/forums/showthread.php?t=134856">Reading List</a>',
 '<a href="http://www.mobileread.com/forums/showthread.php?t=126727">Extract ISBN</a>' and
 '<a href="http://www.mobileread.com/forums/showthread.php?t=134000">Count Pages</a>'
@@ -16,5 +16,13 @@ The official distro channel for this plugin is there: <a href="http://www.mobile
 group</a> for the downloader.  That covers the web application and CLI, too.
 </p>

-The source for this plugin is available
-<a href="http://code.google.com/p/fanficdownloader/source/checkout">here</a>.
+The source for this plugin is available at it's 
+<a href="http://code.google.com/p/fanficdownloader">project home</a>.
+<hr />
+
+<p>
+See the <a href="http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderSupportedsites">list of supported sites</a>.
+</p>
+<p>
+Read the <a href="http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderFAQs">FAQs</a>.
+</p>
@@ -9,16 +9,17 @@ __docformat__ = 'restructuredtext en'

 import traceback, copy

-from PyQt4.Qt import (QDialog, QWidget, QVBoxLayout, QHBoxLayout, QLabel, QLineEdit, QFont,
-                      QTextEdit, QComboBox, QCheckBox, QPushButton, QTabWidget, QVariant)
+from PyQt4.Qt import (QDialog, QWidget, QVBoxLayout, QHBoxLayout, QLabel, QLineEdit, QFont, QWidget,
+                      QTextEdit, QComboBox, QCheckBox, QPushButton, QTabWidget, QVariant, QScrollArea)

 from calibre.gui2 import dynamic, info_dialog
 from calibre.utils.config import JSONConfig
 from calibre.gui2.ui import get_gui

 from calibre_plugins.fanfictiondownloader_plugin.dialogs \
-    import (SKIP, ADDNEW, UPDATE, UPDATEALWAYS, OVERWRITE, OVERWRITEALWAYS,
-             CALIBREONLY,collision_order)
+    import (UPDATE, UPDATEALWAYS, OVERWRITE, collision_order)
+
+from calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters import getConfigSections

 from calibre_plugins.fanfictiondownloader_plugin.common_utils \
    import ( get_library_uuid, KeyboardConfigDialog )
@@ -35,30 +36,48 @@ all_prefs = JSONConfig('plugins/fanfictiondownloader_plugin')
 # Set defaults used by all.  Library specific settings continue to
 # take from here.
 all_prefs.defaults['personal.ini'] = get_resources('plugin-example.ini')
+
 all_prefs.defaults['updatemeta'] = True
+all_prefs.defaults['updatecover'] = False
 all_prefs.defaults['keeptags'] = False
 all_prefs.defaults['urlsfromclip'] = True
 all_prefs.defaults['updatedefault'] = True
 all_prefs.defaults['fileform'] = 'epub'
 all_prefs.defaults['collision'] = OVERWRITE
 all_prefs.defaults['deleteotherforms'] = False
+all_prefs.defaults['adddialogstaysontop'] = False
+all_prefs.defaults['includeimages'] = False
+all_prefs.defaults['lookforurlinhtml'] = False
+
 all_prefs.defaults['send_lists'] = ''
 all_prefs.defaults['read_lists'] = ''
 all_prefs.defaults['addtolists'] = False
 all_prefs.defaults['addtoreadlists'] = False
 all_prefs.defaults['addtolistsonread'] = False
+
+all_prefs.defaults['gcnewonly'] = False
+all_prefs.defaults['gc_site_settings'] = {}
+all_prefs.defaults['allow_gc_from_ini'] = True
+
 all_prefs.defaults['custom_cols'] = {}

 # The list of settings to copy from all_prefs or the previous library
 # when config is called for the first time on a library.
 copylist = ['personal.ini',
            'updatemeta',
+            'updatecover',
            'keeptags',
            'urlsfromclip',
            'updatedefault',
            'fileform',
            'collision',
-            'deleteotherforms']
+            'deleteotherforms',
+            'adddialogstaysontop',
+            'includeimages',
+            'lookforurlinhtml',
+            'gcnewonly',
+            'gc_site_settings',
+            'allow_gc_from_ini']

 # fake out so I don't have to change the prefs calls anywhere.  The
 # Java programmer in me is offended by op-overloading, but it's very
@@ -78,7 +97,8 @@ class PrefsFacade():
                self.all_prefs[libraryid] = dict(self._get_copylist_prefs(self.all_prefs))
            else:
                self.all_prefs[libraryid] = dict(self._get_copylist_prefs(self.all_prefs[self.lastlibid]))
-            self.lastlibid = libraryid
+                
+        self.lastlibid = libraryid
            
        return self.all_prefs[libraryid]

@@ -117,6 +137,10 @@ class ConfigWidget(QWidget):
        self.l = QVBoxLayout()
        self.setLayout(self.l)

+        label = QLabel('<a href="http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderSupportedsites">List of Supported Sites</a> -- <a href="http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderFAQs">FAQs</a>')
+        label.setOpenExternalLinks(True)
+        self.l.addWidget(label)
+
        tab_widget = QTabWidget(self)
        self.l.addWidget(tab_widget)

@@ -126,10 +150,15 @@ class ConfigWidget(QWidget):
        self.personalini_tab = PersonalIniTab(self, plugin_action)
        tab_widget.addTab(self.personalini_tab, 'personal.ini')
        
-        self.list_tab = ListTab(self, plugin_action)
-        tab_widget.addTab(self.list_tab, 'Reading Lists')
+        self.readinglist_tab = ReadingListTab(self, plugin_action)
+        tab_widget.addTab(self.readinglist_tab, 'Reading Lists')
        if 'Reading List' not in plugin_action.gui.iactions:
-            self.list_tab.setEnabled(False)
+            self.readinglist_tab.setEnabled(False)
+
+        self.generatecover_tab = GenerateCoverTab(self, plugin_action)
+        tab_widget.addTab(self.generatecover_tab, 'Generate Cover')
+        if 'Generate Cover' not in plugin_action.gui.iactions:
+            self.generatecover_tab.setEnabled(False)

        self.columns_tab = ColumnsTab(self, plugin_action)
        tab_widget.addTab(self.columns_tab, 'Custom Columns')
@@ -144,20 +173,24 @@ class ConfigWidget(QWidget):
        prefs['fileform'] = unicode(self.basic_tab.fileform.currentText())
        prefs['collision'] = unicode(self.basic_tab.collision.currentText())
        prefs['updatemeta'] = self.basic_tab.updatemeta.isChecked()
+        prefs['updatecover'] = self.basic_tab.updatecover.isChecked()
        prefs['keeptags'] = self.basic_tab.keeptags.isChecked()
        prefs['urlsfromclip'] = self.basic_tab.urlsfromclip.isChecked()
        prefs['updatedefault'] = self.basic_tab.updatedefault.isChecked()
        prefs['deleteotherforms'] = self.basic_tab.deleteotherforms.isChecked()
+        prefs['adddialogstaysontop'] = self.basic_tab.adddialogstaysontop.isChecked()
+        prefs['includeimages'] = self.basic_tab.includeimages.isChecked()
+        prefs['lookforurlinhtml'] = self.basic_tab.lookforurlinhtml.isChecked()

-        if self.list_tab:
+        if self.readinglist_tab:
            # lists
-            prefs['send_lists'] = ', '.join(map( lambda x : x.strip(), filter( lambda x : x.strip() != '', unicode(self.list_tab.send_lists_box.text()).split(','))))
-            prefs['read_lists'] = ', '.join(map( lambda x : x.strip(), filter( lambda x : x.strip() != '', unicode(self.list_tab.read_lists_box.text()).split(','))))
+            prefs['send_lists'] = ', '.join(map( lambda x : x.strip(), filter( lambda x : x.strip() != '', unicode(self.readinglist_tab.send_lists_box.text()).split(','))))
+            prefs['read_lists'] = ', '.join(map( lambda x : x.strip(), filter( lambda x : x.strip() != '', unicode(self.readinglist_tab.read_lists_box.text()).split(','))))
            # print("send_lists: %s"%prefs['send_lists'])
            # print("read_lists: %s"%prefs['read_lists'])
-            prefs['addtolists'] = self.list_tab.addtolists.isChecked()
-            prefs['addtoreadlists'] = self.list_tab.addtoreadlists.isChecked()
-            prefs['addtolistsonread'] = self.list_tab.addtolistsonread.isChecked()
+            prefs['addtolists'] = self.readinglist_tab.addtolists.isChecked()
+            prefs['addtoreadlists'] = self.readinglist_tab.addtoreadlists.isChecked()
+            prefs['addtolistsonread'] = self.readinglist_tab.addtolistsonread.isChecked()

        # personal.ini
        ini = unicode(self.personalini_tab.ini.toPlainText())
@@ -167,6 +200,17 @@ class ConfigWidget(QWidget):
            # if they've removed everything, reset to default.
            prefs['personal.ini'] = get_resources('plugin-example.ini')

+        # Generate Covers tab
+        prefs['gcnewonly'] = self.generatecover_tab.gcnewonly.isChecked()
+        gc_site_settings = {}
+        for (site,combo) in self.generatecover_tab.gc_dropdowns.iteritems():
+            val = unicode(combo.itemData(combo.currentIndex()).toString())
+            if val != 'none':
+                gc_site_settings[site] = val
+                #print("gc_site_settings[%s]:%s"%(site,gc_site_settings[site]))
+        prefs['gc_site_settings'] = gc_site_settings
+        prefs['allow_gc_from_ini'] = self.generatecover_tab.allow_gc_from_ini.isChecked()
+
        # Custom Columns tab
        colsmap = {}
        for (col,combo) in self.columns_tab.custcol_dropdowns.iteritems():
@@ -175,7 +219,7 @@ class ConfigWidget(QWidget):
                colsmap[col] = val
                #print("colsmap[%s]:%s"%(col,colsmap[col]))
        prefs['custom_cols'] = colsmap
-        
+
    def edit_shortcuts(self):
        self.save_settings()
        # Force the menus to be rebuilt immediately, so we have all our actions registered
@@ -234,6 +278,11 @@ class BasicTab(QWidget):
        self.updatemeta.setChecked(prefs['updatemeta'])
        self.l.addWidget(self.updatemeta)

+        self.updatecover = QCheckBox('Update Cover when Updating Metadata?',self)
+        self.updatecover.setToolTip("Update cover image from EPUB when metadata is updated.  (EPUB only.)\nDoesn't go looking for new images on 'Update Calibre Metadata Only'.")
+        self.updatecover.setChecked(prefs['updatecover'])
+        self.l.addWidget(self.updatecover)
+
        self.keeptags = QCheckBox('Keep Existing Tags when Updating Metadata?',self)
        self.keeptags.setToolTip('Existing tags will be kept and any new tags added.\nCompleted and In-Progress tags will be still be updated, if known.\nLast Updated tags will be updated if lastupdate in include_subject_tags.')
        self.keeptags.setChecked(prefs['keeptags'])
@@ -255,6 +304,22 @@ class BasicTab(QWidget):
        self.deleteotherforms.setChecked(prefs['deleteotherforms'])
        self.l.addWidget(self.deleteotherforms)
        
+        self.adddialogstaysontop = QCheckBox("Keep 'Add New from URL(s)' dialog on top?",self)
+        self.adddialogstaysontop.setToolTip("Instructs the OS and Window Manager to keep the 'Add New from URL(s)'\ndialog on top of all other windows.  Useful for dragging URLs onto it.")
+        self.adddialogstaysontop.setChecked(prefs['adddialogstaysontop'])
+        self.l.addWidget(self.adddialogstaysontop)
+
+        # this is a cheat to make it easier for users to realize there's a new include_images features.
+        self.includeimages = QCheckBox("Include images in EPUBs?",self)
+        self.includeimages.setToolTip("Download and include images in EPUB stories.  This is equivalent to adding:\n\n[epub]\ninclude_images:true\nkeep_summary_html:true\nmake_firstimage_cover:true\n\n ...to the top of personal.ini.  Your settings in personal.ini will override this.")
+        self.includeimages.setChecked(prefs['includeimages'])
+        self.l.addWidget(self.includeimages)
+
+        self.lookforurlinhtml = QCheckBox("Search EPUB text for Story URL?",self)
+        self.lookforurlinhtml.setToolTip("Look for first valid story URL inside EPUB text if not found in metadata.\nSomewhat risky, could find wrong URL depending on EPUB content.\nAlso finds and corrects bad ffnet URLs from ficsaver.com files.")
+        self.lookforurlinhtml.setChecked(prefs['lookforurlinhtml'])
+        self.l.addWidget(self.lookforurlinhtml)
+
        self.l.insertStretch(-1)
        
    def set_collisions(self):
@@ -340,7 +405,7 @@ class ShowDefaultsIniDialog(QDialog):
        self.ok_button.clicked.connect(self.hide)
        self.l.addWidget(self.ok_button)
        
-class ListTab(QWidget):
+class ReadingListTab(QWidget):

    def __init__(self, parent_dialog, plugin_action):
        self.parent_dialog = parent_dialog
@@ -400,6 +465,75 @@ class ListTab(QWidget):
            
        self.l.insertStretch(-1)
        
+class GenerateCoverTab(QWidget):
+
+    def __init__(self, parent_dialog, plugin_action):
+        self.parent_dialog = parent_dialog
+        self.plugin_action = plugin_action
+        QWidget.__init__(self)
+        
+        self.l = QVBoxLayout()
+        self.setLayout(self.l)
+
+        try:
+            gc_plugin = plugin_action.gui.iactions['Generate Cover']
+            gc_settings = gc_plugin.get_saved_setting_names()
+        except KeyError:
+            gc_settings= []
+            
+        label = QLabel('The Generate Cover plugin can create cover images for books using various metadata and configurations.  If you have GC installed, FFDL can run GC on new downloads and metadata updates.  Pick a GC setting by site or Default.')
+        label.setWordWrap(True)
+        self.l.addWidget(label)
+        self.l.addSpacing(5)
+
+        scrollable = QScrollArea()
+        scrollcontent = QWidget()
+        scrollable.setWidget(scrollcontent)
+        scrollable.setWidgetResizable(True)
+        self.l.addWidget(scrollable)
+
+        self.sl = QVBoxLayout()
+        scrollcontent.setLayout(self.sl)
+        
+        self.gc_dropdowns = {}
+
+        sitelist = getConfigSections()
+        sitelist.sort()
+        sitelist.insert(0,u"Default")
+        for site in sitelist:
+            horz = QHBoxLayout()
+            label = QLabel(site)
+            if site == u"Default":
+                s = "On Metadata update, run Generate Cover with this setting, if not selected for specific site."
+            else:
+                s = "On Metadata update, run Generate Cover with this setting for site (%s)."%site
+
+            label.setToolTip(s)
+            horz.addWidget(label)
+            dropdown = QComboBox(self)
+            dropdown.setToolTip(s)
+            dropdown.addItem('',QVariant('none'))
+            for setting in gc_settings:
+                dropdown.addItem(setting,QVariant(setting))
+            self.gc_dropdowns[site] = dropdown
+            if site in prefs['gc_site_settings']:
+                dropdown.setCurrentIndex(dropdown.findData(QVariant(prefs['gc_site_settings'][site])))
+
+            horz.addWidget(dropdown)
+            self.sl.addLayout(horz)
+        
+        self.gcnewonly = QCheckBox("Run Generate Cover Only on New Books",self)
+        self.gcnewonly.setToolTip("Default is to run GC any time the calibre metadata is updated.")
+        self.gcnewonly.setChecked(prefs['gcnewonly'])
+        self.l.addWidget(self.gcnewonly)
+
+        self.allow_gc_from_ini = QCheckBox('Allow generate_cover_settings from personal.ini to override.',self)
+        self.allow_gc_from_ini.setToolTip("The INI parameter generate_cover_settings allows you to choose a GC setting based on metadata rather than site,\nbut it's much more complex.  generate_cover_settings is ignored when this is off.")
+        self.allow_gc_from_ini.setChecked(prefs['allow_gc_from_ini'])
+        self.l.addWidget(self.allow_gc_from_ini)
+            
+        self.l.insertStretch(-1)
+        
 class OtherTab(QWidget):

    def __init__(self, parent_dialog, plugin_action):
@@ -1,30 +0,0 @@
-#!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
-from __future__ import (unicode_literals, division, absolute_import,
-                        print_function)
-
-__license__   = 'GPL v3'
-__copyright__ = '2012, Jim Miller'
-__docformat__ = 'restructuredtext en'
-
-from zipfile import ZipFile
-
-from xml.dom.minidom import parseString
-
-def get_dcsource(inputio):
-    epub = ZipFile(inputio, 'r')
-
-    ## Find the .opf file.
-    container = epub.read("META-INF/container.xml")
-    containerdom = parseString(container)
-    rootfilenodelist = containerdom.getElementsByTagName("rootfile")
-    rootfilename = rootfilenodelist[0].getAttribute("full-path")
-
-    metadom = parseString(epub.read(rootfilename))
-    firstmetadom = metadom.getElementsByTagName("metadata")[0]
-    try:
-        source=firstmetadom.getElementsByTagName("dc:source")[0].firstChild.data.encode("utf-8")
-    except:
-        source=None
-
-    return source
@@ -13,7 +13,7 @@ from PyQt4 import QtGui
 from PyQt4.Qt import (QDialog, QTableWidget, QMessageBox, QVBoxLayout, QHBoxLayout, QGridLayout,
                      QPushButton, QProgressDialog, QString, QLabel, QCheckBox, QIcon, QTextCursor,
                      QTextEdit, QLineEdit, QInputDialog, QComboBox, QClipboard, QVariant,
-                      QProgressDialog, QTimer, QDialogButtonBox, QPixmap, Qt,QAbstractItemView )
+                      QProgressDialog, QTimer, QDialogButtonBox, QPixmap, Qt, QAbstractItemView )

 from calibre.gui2 import error_dialog, warning_dialog, question_dialog, info_dialog
 from calibre.gui2.dialogs.confirm_delete import confirm
@@ -60,9 +60,8 @@ class DroppableQTextEdit(QTextEdit):
            return QTextEdit.canInsertFromMimeData(self,source)

    def insertFromMimeData(self, source):
-        if source.hasUrls():
-            for u in source.urls():
-                self.append(u.toString())
+        if source.hasText():
+            self.append(source.text())
        else:
            return QTextEdit.insertFromMimeData(self, source)
                            
@@ -72,6 +71,9 @@ class AddNewDialog(SizePersistedDialog):
        SizePersistedDialog.__init__(self, gui, 'FanFictionDownLoader plugin:add new dialog')
        self.gui = gui

+        if prefs['adddialogstaysontop']:
+            QDialog.setWindowFlags ( self, Qt.Dialog|Qt.WindowStaysOnTopHint )
+        
        self.setMinimumWidth(300)
        self.l = QVBoxLayout()
        self.setLayout(self.l)
@@ -157,25 +159,40 @@ class AddNewDialog(SizePersistedDialog):

    def get_urlstext(self):
        return unicode(self.url.toPlainText())
-        
+
+
+class FakeLineEdit():
+    def __init__(self):
+        pass
+    
+    def text(self):
+        pass
+    
 class UserPassDialog(QDialog):
    '''
    Need to collect User/Pass for some sites.
    '''
-    def __init__(self, gui, site):
+    def __init__(self, gui, site, exception=None):
        QDialog.__init__(self, gui)
        self.gui = gui
        self.status=False
-        self.setWindowTitle('User/Password')

        self.l = QGridLayout()
        self.setLayout(self.l)

-        self.l.addWidget(QLabel("%s requires you to login to download this story."%site),0,0,1,2)
+        if exception.passwdonly:
+            self.setWindowTitle('Password')
+            self.l.addWidget(QLabel("Author requires a password for this story(%s)."%exception.url),0,0,1,2)
+            # user isn't used, but it's easier to still have it for
+            # post processing.
+            self.user = FakeLineEdit()
+        else:
+            self.setWindowTitle('User/Password')
+            self.l.addWidget(QLabel("%s requires you to login to download this story."%site),0,0,1,2)
        
-        self.l.addWidget(QLabel("User:"),1,0)
-        self.user = QLineEdit(self)
-        self.l.addWidget(self.user,1,1)
+            self.l.addWidget(QLabel("User:"),1,0)
+            self.user = QLineEdit(self)
+            self.l.addWidget(self.user,1,1)
   
        self.l.addWidget(QLabel("Password:"),2,0)
        self.passwd = QLineEdit(self)
@@ -7,14 +7,19 @@ __license__   = 'GPL v3'
 __copyright__ = '2012, Jim Miller'
 __docformat__ = 'restructuredtext en'

-import time, os, copy, threading
+import time, os, copy, threading, re
 from ConfigParser import SafeConfigParser
 from StringIO import StringIO
 from functools import partial
 from datetime import datetime
+from string import Template

 from PyQt4.Qt import (QApplication, QMenu, QToolButton)

+from PyQt4.Qt import QPixmap, Qt
+from PyQt4.QtCore import QBuffer
+
+
 from calibre.ptempfile import PersistentTemporaryFile, PersistentTemporaryDirectory, remove_dir
 from calibre.ebooks.metadata import MetaInformation, authors_to_string
 from calibre.ebooks.metadata.meta import get_metadata
@@ -30,8 +35,8 @@ from calibre_plugins.fanfictiondownloader_plugin.common_utils import (set_plugin
                                         create_menu_action_unique, get_library_uuid)

 from calibre_plugins.fanfictiondownloader_plugin.fanficdownloader import adapters, writers, exceptions
-from calibre_plugins.fanfictiondownloader_plugin.epubmerge import doMerge
-from calibre_plugins.fanfictiondownloader_plugin.dcsource import get_dcsource
+from calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.htmlcleanup import stripHTML
+from calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.epubutils import get_dcsource, get_dcsource_chaptercount, get_story_url_from_html

 from calibre_plugins.fanfictiondownloader_plugin.config import (prefs, permitted_values)
 from calibre_plugins.fanfictiondownloader_plugin.dialogs import (
@@ -93,6 +98,8 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
        # are not found in the zip file will result in null QIcons.
        icon = get_icon('images/icon.png')

+        #self.qaction.setText('FFDL')
+        
        # The qaction is automatically created from the action_spec defined
        # above
        self.qaction.setIcon(icon)
@@ -103,6 +110,9 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
        # Assign our menu to this action
        self.menu = QMenu(self.gui)
        self.old_actions_unique_map = {}
+        # menu_actions is just to keep a live reference to the menu
+        # items to prevent GC removing it.
+        self.menu_actions = []
        self.qaction.setMenu(self.menu)
        self.menu.aboutToShow.connect(self.about_to_show_menu)

@@ -129,6 +139,7 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
            do_user_config = self.interface_action_base_plugin.do_user_config
            self.menu.clear()
            self.actions_unique_map = {}
+            self.menu_actions = []
            self.add_action = self.create_menu_item_ex(self.menu, '&Add New from URL(s)', image='plus.png',
                                                       unique_name='Add New FanFiction Book(s) from URL(s)',
                                                       shortcut_name='Add New FanFiction Book(s) from URL(s)',
@@ -194,9 +205,6 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
                                                           shortcut_name='About FanFictionDownLoader',
                                                           triggered=self.about)
            
-            # self.update_action.setEnabled( len(self.gui.library_view.get_selected_ids()) > 0 )
-            # self.get_list_action.setEnabled( len(self.gui.library_view.get_selected_ids()) > 0 )
-
            # Before we finalize, make sure we delete any actions for menus that are no longer displayed
            for menu_id, unique_name in self.old_actions_unique_map.iteritems():
                if menu_id not in self.actions_unique_map:
@@ -217,13 +225,14 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
        
        text = get_resources('about.txt')
        AboutDialog(self.gui,self.qaction.icon(),self.version + text).exec_()
-        
+
    def create_menu_item_ex(self, parent_menu, menu_text, image=None, tooltip=None,
                           shortcut=None, triggered=None, is_checked=None, shortcut_name=None,
                           unique_name=None):
        ac = create_menu_action_unique(self, parent_menu, menu_text, image, tooltip,
                                       shortcut, triggered, is_checked, shortcut_name, unique_name)
        self.actions_unique_map[ac.calibre_shortcut_unique_name] = ac.calibre_shortcut_unique_name
+        self.menu_actions.append(ac)
        return ac

    def plugin_button(self):
@@ -236,7 +245,6 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
        if len(self.gui.library_view.get_selected_ids()) > 0 and \
                (prefs['addtolists'] or prefs['addtoreadlists']) :
            self._update_reading_lists(self.gui.library_view.get_selected_ids(),add)
-        #self.gui.library_view.model().refresh_ids(self.gui.library_view.get_selected_ids())

    def get_list_urls(self):
        if len(self.gui.library_view.get_selected_ids()) > 0:
@@ -408,18 +416,25 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
        ffdlconfig = SafeConfigParser()
        ffdlconfig.readfp(StringIO(get_resources("plugin-defaults.ini")))
        ffdlconfig.readfp(StringIO(prefs['personal.ini']))
-        adapter = adapters.getAdapter(ffdlconfig,url)
+        adapter = adapters.getAdapter(ffdlconfig,url,fileform)

        options['personal.ini'] = prefs['personal.ini']
+        if prefs['includeimages']:
+            # this is a cheat to make it easier for users.
+            options['personal.ini'] = '''[epub]
+include_images:true
+keep_summary_html:true
+make_firstimage_cover:true
+''' + options['personal.ini']

        ## three tries, that's enough if both user/pass & is_adult needed,
        ## or a couple tries of one or the other
        for x in range(0,2):
            try:
                adapter.getStoryMetadataOnly()
-            except exceptions.FailedToLogin:
+            except exceptions.FailedToLogin, f:
                print("Login Failed, Need Username/Password.")
-                userpass = UserPassDialog(self.gui,url)
+                userpass = UserPassDialog(self.gui,url,f)
                userpass.exec_() # exec_ will make it act modal
                if userpass.status:
                    adapter.username = userpass.user.text()
@@ -439,8 +454,8 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
        book['title'] = story.getMetadata("title", removeallentities=True)
        book['author_sort'] = book['author'] = story.getMetadata("author", removeallentities=True)
        book['publisher'] = story.getMetadata("site")
-        book['tags'] = writer.getTags()
-        book['comments'] = story.getMetadata("description") #, removeallentities=True) comments handles entities better.
+        book['tags'] = writer.getTags() # getTags could be moved up into adapter now.  Adapter didn't used to know the fileform
+        book['comments'] = stripHTML(story.getMetadata("description")) #, removeallentities=True) comments handles entities better.
        book['series'] = story.getMetadata("series")
        
        # adapter.opener is the element with a threadlock.  But del
@@ -497,8 +512,11 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
            if len(identicalbooks) > 1:
                raise NotGoingToDownload("More than one identical book--can't tell which to update/overwrite.","minusminus.png")

+            ## changed: add new book when CALIBREONLY if none found.
            if collision == CALIBREONLY and not identicalbooks:
-                raise NotGoingToDownload("Not updating Calibre Metadata, no existing book to update.","search_delete_saved.png")
+                collision = ADDNEW
+                options['collision'] = ADDNEW
+            #     raise NotGoingToDownload("Not updating Calibre Metadata, no existing book to update.","search_delete_saved.png")

            if len(identicalbooks)>0:
                book_id = identicalbooks.pop()
@@ -506,7 +524,7 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
                book['icon'] = 'edit-redo.png'

        if book_id != None and collision != ADDNEW:
-            if options['collision'] in (CALIBREONLY):
+            if collision in (CALIBREONLY):
                book['comment'] = 'Metadata collected.'
                # don't need temp file created below.
                return
@@ -517,13 +535,9 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
                # 'book' can exist without epub.  If there's no existing epub,
                # let it go and it will download it.
                if db.has_format(book_id,fileform,index_is_id=True):
-                    toupdateio = StringIO()
-                    (epuburl,chaptercount) = doMerge(toupdateio,
-                                                     [StringIO(db.format(book_id,'EPUB',
-                                                                              index_is_id=True))],
-                                                     titlenavpoints=False,
-                                                     striptitletoc=True,
-                                                     forceunique=False)
+                    (epuburl,chaptercount) = \
+                        get_dcsource_chaptercount(StringIO(db.format(book_id,'EPUB',
+                                                                     index_is_id=True)))
                    urlchaptercount = int(story.getMetadata('numChapters'))
                    if chaptercount == urlchaptercount:
                        if collision == UPDATE:
@@ -630,17 +644,19 @@ class FanFictionDownLoaderPlugin(InterfaceAction):

        if options['collision'] == CALIBREONLY or \
                (options['updatemeta'] and book['good']):
-            self._update_metadata(db, book['calibre_id'], book, mi)        
+            self._update_metadata(db, book['calibre_id'], book, mi, options)

    def _update_books_completed(self, book_list, options={}):
        
        add_list = filter(lambda x : x['good'] and x['added'], book_list)
+        add_ids = [ x['calibre_id'] for x in add_list ]
        update_list = filter(lambda x : x['good'] and not x['added'], book_list)
        update_ids = [ x['calibre_id'] for x in update_list ]
        
        if len(add_list):
            ## even shows up added to searchs.  Nice.
            self.gui.library_view.model().books_added(len(add_list))
+            self.gui.library_view.model().refresh_ids(add_ids)

        if update_ids:
            self.gui.library_view.model().refresh_ids(update_ids)
@@ -649,6 +665,9 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
        self.gui.library_view.model().current_changed(current, self.previous)
        self.gui.tags_view.recount()
        
+        if self.gui.cover_flow:
+            self.gui.cover_flow.dataChanged()
+        
        self.gui.status_bar.show_message(_('Finished Adding/Updating %d books.'%(len(update_list) + len(add_list))), 3000)
            
        if len(update_list) + len(add_list) != len(book_list):
@@ -729,7 +748,7 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
                
        return book_id

-    def _update_metadata(self, db, book_id, book, mi):
+    def _update_metadata(self, db, book_id, book, mi, options):
        if prefs['keeptags']:
            old_tags = db.get_tags(book_id)
            # remove old Completed/In-Progress only if there's a new one.
@@ -748,7 +767,19 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
            oldmi = db.get_metadata(book_id,index_is_id=True)
            if not oldmi.languages:
                mi.languages=['eng']
-                
+
+        if options['fileform'] == 'epub' and prefs['updatecover']:
+            existingepub = db.format(book_id,'EPUB',index_is_id=True, as_file=True)
+            epubmi = get_metadata(existingepub,'EPUB')
+            if epubmi.cover_data[1] is not None:
+                db.set_cover(book_id, epubmi.cover_data[1])
+
+        # set author link if found.  All current adapters have authorUrl.
+        if 'authorUrl' in book['all_metadata']:
+            autid=db.get_author_id(book['author'])
+            db.set_link_field_for_author(autid, unicode(book['all_metadata']['authorUrl']),
+                                         commit=False, notify=False)
+            
        db.set_metadata(book_id,mi)

        # do configured column updates here.
@@ -762,7 +793,8 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
                print("%s not an existing column, skipping."%col)
                continue
            coldef = custom_columns[col]
-            if not meta.startswith('status-') and meta not in book['all_metadata']:
+            if not meta.startswith('status-') and meta not in book['all_metadata'] or \
+                    meta.startswith('status-') and 'status' not in book['all_metadata']:
                print("No value for %s, skipping."%meta)
                continue
            if meta not in permitted_values[coldef['datatype']]:
@@ -780,8 +812,48 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
                if meta == 'status-I':
                    val = book['all_metadata']['status'] == 'In-Progress'
                db.set_custom(book_id, val, label=label, commit=False)
-                
+
        db.commit()
+        
+        print("book['added']:%s"%book['added'])
+        if 'Generate Cover' in self.gui.iactions and (book['added'] or not prefs['gcnewonly']):
+            gc_plugin = self.gui.iactions['Generate Cover']
+            setting_name = None
+            if prefs['allow_gc_from_ini']:
+                ffdlconfig = SafeConfigParser()
+                ffdlconfig.readfp(StringIO(get_resources("plugin-defaults.ini")))
+                ffdlconfig.readfp(StringIO(prefs['personal.ini']))
+                adapter = adapters.getAdapter(ffdlconfig,book['url'],options['fileform'])
+
+                # template => regexp to match => GC Setting to use.
+                # generate_cover_settings:
+                # ${category} => Buffy:? the Vampire Slayer => Buffy
+                for line in adapter.getConfig('generate_cover_settings').splitlines():
+                    if "=>" in line:
+                        (template,regexp,setting) = map( lambda x: x.strip(), line.split("=>") )
+                        value = Template(template).substitute(book['all_metadata']).encode('utf8')
+                        print("%s(%s) => %s => %s"%(template,value,regexp,setting))
+                        if re.search(regexp,value):
+                            setting_name = setting
+                            break
+
+                if setting_name:
+                    print("Generate Cover Setting from generate_cover_settings(%s)"%line)
+                    if setting_name not in gc_plugin.get_saved_setting_names():
+                        print("GC Name %s not found, discarding! (check personal.ini for typos)"%setting_name)
+                        setting_name = None
+                
+            if not setting_name and book['all_metadata']['site'] in prefs['gc_site_settings']:
+                setting_name =  prefs['gc_site_settings'][book['all_metadata']['site']]
+
+            if not setting_name and 'Default' in prefs['gc_site_settings']:
+                setting_name =  prefs['gc_site_settings']['Default']
+                
+            if setting_name:
+                print("Running Generate Cover with settings %s."%setting_name)
+                realmi = db.get_metadata(book_id, index_is_id=True)
+                gc_plugin.generate_cover_for_book(realmi,saved_setting_name=setting_name)
+        

    def _get_clean_reading_lists(self,lists):
        if lists == None or lists.strip() == "" :
@@ -900,13 +972,6 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
        book['added'] = False
        
        return book
-        
-    
-    # def _convert_calibre_ids_to_books(self, db, ids):
-    #     books = []
-    #     for book_id in ids:
-    #         books.append(self._convert_calibre_id_to_book(db,book_id))
-    #     return books
            
    def _populate_book_from_calibre_id(self, book, db=None):
        mi = db.get_metadata(book['calibre_id'], index_is_id=True)
@@ -953,8 +1018,12 @@ class FanFictionDownLoaderPlugin(InterfaceAction):
                if 'url' in identifiers:
                    #print("url from epub:"+identifiers['url'].replace('|',':'))
                    return identifiers['url'].replace('|',':')
-                # look for dc:source
-                return get_dcsource(existingepub)
+                # look for dc:source first, then scan HTML if 
+                link = get_dcsource(existingepub)
+                if link:
+                    return link
+                elif prefs['lookforurlinhtml']:
+                    return get_story_url_from_html(existingepub,self._is_good_downloader_url)
        return None

    def _is_good_downloader_url(self,url):
@@ -12,10 +12,7 @@ import time, os, traceback

 from ConfigParser import SafeConfigParser
 from StringIO import StringIO
-#from itertools import izip
-#from threading import Event

-#from calibre.gui2.convert.single import sort_formats_by_preference
 from calibre.utils.ipc.server import Server
 from calibre.utils.ipc.job import ParallelJob
 from calibre.utils.logging import Log
@@ -23,7 +20,7 @@ from calibre.utils.logging import Log
 from calibre_plugins.fanfictiondownloader_plugin.dialogs import (NotGoingToDownload,
    OVERWRITE, OVERWRITEALWAYS, UPDATE, UPDATEALWAYS, ADDNEW, SKIP, CALIBREONLY)
 from calibre_plugins.fanfictiondownloader_plugin.fanficdownloader import adapters, writers, exceptions
-from calibre_plugins.fanfictiondownloader_plugin.epubmerge import doMerge
+from calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.epubutils import get_update_data

 # ------------------------------------------------------------------------------
 #
@@ -110,7 +107,7 @@ def do_download_for_worker(book,options):
        ffdlconfig.readfp(StringIO(get_resources("plugin-defaults.ini")))
        ffdlconfig.readfp(StringIO(options['personal.ini']))
        
-        adapter = adapters.getAdapter(ffdlconfig,book['url'])
+        adapter = adapters.getAdapter(ffdlconfig,book['url'],options['fileform'])
        adapter.is_adult = book['is_adult'] 
        adapter.username = book['username'] 
        adapter.password = book['password']
@@ -135,39 +132,17 @@ def do_download_for_worker(book,options):
        ## checks were done earlier, just update it.
        elif 'epub_for_update' in book and options['collision'] in (UPDATE, UPDATEALWAYS):
            
+            # update now handled by pre-populating the old images and
+            # chapters in the adapter rather than merging epubs.
            urlchaptercount = int(story.getMetadata('numChapters'))
-            ## First, get existing epub with titlepage and tocpage stripped.
-            updateio = StringIO()
-            (epuburl,chaptercount) = doMerge(updateio,
-                                             [book['epub_for_update']],
-                                             titlenavpoints=False,
-                                             striptitletoc=True,
-                                             forceunique=False)
+            (url,chaptercount,
+             adapter.oldchapters,
+             adapter.oldimgs) = get_update_data(book['epub_for_update'])
+
            print("Do update - epub(%d) vs url(%d)" % (chaptercount, urlchaptercount))
            print("write to %s"%outfile)

-            ## Get updated title page/metadata by itself in an epub.
-            ## Even if the title page isn't included, this carries the metadata.
-            titleio = StringIO()
-            writer.writeStory(outstream=titleio,metaonly=True)
-
-            newchaptersio = None
-            if urlchaptercount > chaptercount :
-                ## Go get the new chapters
-                newchaptersio = StringIO()
-                adapter.setChaptersRange(chaptercount+1,urlchaptercount)
-
-                adapter.config.set("overrides",'include_tocpage','false')
-                adapter.config.set("overrides",'include_titlepage','false')
-                writer.writeStory(outstream=newchaptersio)
-
-            ## Merge the three epubs together.
-            doMerge(outfile,
-                    [titleio,updateio,newchaptersio],
-                    fromfirst=True,
-                    titlenavpoints=False,
-                    striptitletoc=False,
-                    forceunique=False)
+            writer.writeStory(outfilename=outfile, forceOverwrite=True)
            
            book['comment'] = 'Update %s completed, added %s chapters for %s total.'%\
                (options['fileform'],(urlchaptercount-chaptercount),urlchaptercount)
@@ -112,9 +112,9 @@ zip_output: false
 ## Can include directories.  .zip will be added if not in name somewhere
 zip_filename: ${title}-${siteabbrev}_${storyId}${formatext}.zip

-## Normally, try to make the output file name 'safe' by removing
-## invalid filename chars.  Applies to both output_filename &
-## zip_filename.  
+## Normally, try to make the filenames 'safe' by removing invalid
+## filename chars.  Applies to default_cover_image, output_filename &
+## zip_filename.
 allow_unsafe_filename: false

 ## entries to make epub subjects and calibre tags
@@ -126,7 +126,6 @@ extratags: FanFiction

 ## number of seconds to sleep between calls to the story site.  May by
 ## useful if pulling large numbers of stories or if the site is slow.
-## Primarily for commandline.
 #slow_down_sleep_time:0.5

 ## For use only with stand-alone CLI version--run a command on the
@@ -148,6 +147,11 @@ extratags: FanFiction
 # (.*)Great(.*)=>\1Moderate\2
 # .*-Centered=>

+## Some readers don't show horizontal rule (<hr />) tags correctly.
+## This replaces them all with a centered '* * *'.  (Note centering
+## doesn't work on some devices either.)
+#replace_hr: false
+
 ## Each output format has a section that overrides [defaults]
 [html]

@@ -231,6 +235,57 @@ output_css:
 .u        {text-decoration: underline;}
 .bold     {font-weight: bold;}

+## include images from img tags in the body and summary of
+## stories.  Images will be converted to jpg for size if possible.
+#include_images:false
+
+## If not set, the summary will have all html stripped for safety.
+## Both this and include_images must be true to get images in the
+## summary.
+#keep_summary_html:false
+
+## If set, the first image found will be made the cover image.  If
+## keep_summary_html is true, any images in summary will be before any
+## in chapters.
+#make_firstimage_cover: false
+
+## If set, the epub will never have a cover, even include_images is on
+## and the site has specific cover images.
+#never_make_cover: false
+
+## If set, and there isn't already a cover image from the adapter or
+## from make_firstimage_cover, this image will be made the cover.
+## It can be either a 'file:' or 'http:' url.
+## Note that if you enable make_firstimage_cover in [epub], but want
+## to use default_cover_image for a specific site, use the site:format
+## section, for example: [www.ficwad.com:epub]
+## default_cover_image is a python string Template string with
+## ${title}, ${author} etc, same as titlepage_entries. Unless
+## allow_unsafe_filename is true, invalid filename chars will be
+## removed from metadata fields
+#default_cover_image:file:///C:/Users/username/Desktop/nook/images/icon.png
+#default_cover_image:file:///C:/Users/username/Desktop/nook/images/${title}/icon.png
+#default_cover_image:http://www.somesite.com/someimage.gif
+
+## some sites include images that we don't ever want becoming the
+## cover image.  This lets you exclude them.
+#cover_exclusion_regexp:/stories/999/images/.*?_trophy.png
+
+## Resize images down to width, height, preserving aspect ratio.
+## Nook size, with margin.
+image_max_size: 580, 725
+
+## Change image to grayscale, if graphics library allows, to save
+## space.
+#grayscale_images: false
+
+## if the <img> tag doesn't have a div or a p around it, nook gets
+## confused and displays it on every page after that under the text
+## for the rest of the chapter.  I doubt adding a div around the img
+## will break any other readers, but in case it does, the fix can be
+## turned off.
+nook_img_fix:true
+
 [mobi]
 ## mobi TOC cannot be turned off right now.
 #include_tocpage: true
@@ -251,8 +306,95 @@ extratags: FanFiction,Testing,Text
 [test1.com:html]
 extratags: FanFiction,Testing,HTML

+[castlefans.org]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+[fanfiction.mugglenet.com]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+[fanfiction.portkey.org]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[fanfiction.tenhawkpresents.com]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+[nfacommunity.com]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+[thequidditchpitch.org]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.adastrafanfic.com]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.archiveofourown.org]
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
 [www.fanfiction.net]

+[www.ficbook.net]
+
+[www.fictionalley.org]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+## fictionally.org storyIds are not unique.  Combine with authorId.
+output_filename: ${title}-${siteabbrev}_${authorId}_${storyId}${formatext}
+
 [www.fictionpress.com]
 ## Clear FanFiction from defaults, fictionpress.com is original fiction.
 extratags:
@@ -265,6 +407,91 @@ extratags:
 #username:YourName
 #password:yourpassword

+[www.fimfiction.net]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+## fimfiction.net stories can be locked requiring individual
+## passwords.  If fail_on_password is set, the downloader will fail
+## when a password is required rather than prompting every time.
+#fail_on_password: false
+
+[www.harrypotterfanfiction.com]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.hpfandom.net]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.libraryofmoria.com]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.mediaminer.org]
+
+[www.midnightwhispers.ca]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+## some sites include images that we don't ever want becoming the
+## cover image.  This lets you exclude them.
+cover_exclusion_regexp:/stories/999/images/.*?_trophy.png
+
+[www.potionsandsnitches.net]
+
+[www.siye.co.uk]
+
+[www.squidge.org/peja]
+# www.squidge.org/peja calls it Fandom <shrug>
+category_label:Fandom
+# Remove numWords -- www.squidge.org/peja word counts are inaccurate
+titlepage_entries: series,category,genre,language,characters,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,site,description
+
+[www.squidge.org/peja:txt]
+## Add URLs since there aren't links.
+# Remove numWords -- www.squidge.org/peja word counts are inaccurate
+titlepage_entries: series,category,genre,language,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,site,storyUrl, authorUrl, description
+
+[www.thewriterscoffeeshop.com]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+## thewriterscoffeeshop.com (ab)uses series as personal reading lists.
+collect_series: false
+
+[www.tthfanfic.org]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+## tth is a little unusual--it doesn't require user/pass, but the site
+## keeps track of which chapters you've read and won't send another
+## update until it thinks you're up to date.  This way, on download,
+## it thinks you're up to date.
+#username:YourName
+#password:yourpassword
+
 [www.twilighted.net]
 ## Some sites require login (or login for some rated stories) The
 ## program can prompt you, or you can save it in config.  In
@@ -289,71 +516,12 @@ collect_series: false

 [www.whofic.com]

-[www.mediaminer.org]
-
-[www.thewriterscoffeeshop.com]
-## Some sites require login (or login for some rated stories) The
-## program can prompt you, or you can save it in config.  In
-## commandline version, this should go in your personal.ini, not
-## defaults.ini.
-#username:YourName
-#password:yourpassword
-
-## Some sites also require the user to confirm they are adult for
-## adult content.  In commandline version, this should go in your
-## personal.ini, not defaults.ini.
-#is_adult:true
-
-## thewriterscoffeeshop.com (ab)uses series as personal reading lists.
-collect_series: false
-
-[www.ficwad.com]
-## Some sites require login (or login for some rated stories) The
-## program can prompt you, or you can save it in config.  In
-## commandline version, this should go in your personal.ini, not
-## defaults.ini.
-#username:YourName
-#password:yourpassword
-
-[www.adastrafanfic.com]
+[www.wraithbait.com]
 ## Some sites do not require a login, but do require the user to
 ## confirm they are adult for adult content.  In commandline version,
 ## this should go in your personal.ini, not defaults.ini.
 #is_adult:true

-[www.fictionalley.org]
-## Some sites do not require a login, but do require the user to
-## confirm they are adult for adult content.  In commandline version,
-## this should go in your personal.ini, not defaults.ini.
-#is_adult:true
-
-## fictionally.org storyIds are not unique.  Combine with authorId.
-output_filename: ${title}-${siteabbrev}_${authorId}_${storyId}${formatext}
-
-[www.harrypotterfanfiction.com]
-## Some sites do not require a login, but do require the user to
-## confirm they are adult for adult content.  In commandline version,
-## this should go in your personal.ini, not defaults.ini.
-#is_adult:true
-
-[www.fimfiction.net]
-## Some sites do not require a login, but do require the user to
-## confirm they are adult for adult content.  In commandline version,
-## this should go in your personal.ini, not defaults.ini.
-#is_adult:true
-
-[www.tthfanfic.org]
-## Some sites do not require a login, but do require the user to
-## confirm they are adult for adult content.  In commandline version,
-## this should go in your personal.ini, not defaults.ini.
-#is_adult:true
-## tth is a little unusual--it doesn't require user/pass, but the site
-## keeps track of which chapters you've read and won't send another
-## update until it thinks you're up to date.  This way, on download,
-## it thinks you're up to date.
-#username:YourName
-#password:yourpassword
-
 [overrides]
 ## It may sometimes be useful to override all of the specific format,
 ## site and site:format sections in your private configuration.  For
@@ -25,19 +25,16 @@ from StringIO import StringIO
 from optparse import OptionParser      
 import getpass
 import string
-
+import ConfigParser
 from subprocess import call

-from epubmerge import doMerge
+from fanficdownloader import adapters,writers,exceptions
+from fanficdownloader.epubutils import get_dcsource_chaptercount, get_update_data

 if sys.version_info < (2, 5):
    print "This program requires Python 2.5 or newer."
    sys.exit(1)

-from fanficdownloader import adapters,writers,exceptions
-
-import ConfigParser
-
 def writeStory(config,adapter,writeformat,metaonly=False,outstream=None):
    writer = writers.getWriter(writeformat,config,adapter)
    writer.writeStory(outstream=outstream,metaonly=metaonly)
@@ -66,7 +63,7 @@ def main():
                     help="Retrieve metadata and stop.  Or, if --update-epub, update metadata title page only.",)
   parser.add_option("-u", "--update-epub",
                     action="store_true", dest="update",
-                     help="Update an existing epub with new chapter, give epub filename instead of storyurl.  Not compatible with inserted TOC.",)
+                     help="Update an existing epub with new chapter, give epub filename instead of storyurl.",)
   parser.add_option("--force",
                     action="store_true", dest="force",
                     help="Force overwrite or update of an existing epub, download and overwrite all chapters.",)
@@ -116,29 +113,37 @@ def main():
   try:
       ## Attempt to update an existing epub.
       if options.update:
-           updateio = StringIO()
-           (url,chaptercount) = doMerge(updateio,
-                                        args,
-                                        titlenavpoints=False,
-                                        striptitletoc=True,
-                                        forceunique=False)
+           (url,chaptercount) = get_dcsource_chaptercount(args[0])
           print "Updating %s, URL: %s" % (args[0],url)
           output_filename = args[0]
           config.set("overrides","output_filename",args[0])
       else:
           url = args[0]

-       adapter = adapters.getAdapter(config,url)
+       adapter = adapters.getAdapter(config,url,options.format)
+
+       ## Check for include_images and absence of PIL, give warning.
+       if adapter.getConfig('include_images'):
+           try:
+               import Image
+           except:
+               print "You have include_images enabled, but Python Image Library(PIL) isn't found.\nImages will be included full size in original format.\nContinue? (y/n)?"
+               if not sys.stdin.readline().strip().lower().startswith('y'):
+                   return
+               

       ## three tries, that's enough if both user/pass & is_adult needed,
       ## or a couple tries of one or the other
       for x in range(0,2):
           try:
               adapter.getStoryMetadataOnly()
-           except exceptions.FailedToLogin:
-               print "Login Failed, Need Username/Password."
-               sys.stdout.write("Username: ")
-               adapter.username = sys.stdin.readline().strip()
+           except exceptions.FailedToLogin, f:
+               if f.passwdonly:
+                   print "Story requires a password."
+               else:
+                   print "Login Failed, Need Username/Password."
+                   sys.stdout.write("Username: ")
+                   adapter.username = sys.stdin.readline().strip()
               adapter.password = getpass.getpass(prompt='Password: ')
               #print("Login: `%s`, Password: `%s`" % (adapter.username, adapter.password))
           except exceptions.AdultCheckRequired:
@@ -155,40 +160,17 @@ def main():
               print "%s contains %d chapters, more than source: %d." % (args[0],chaptercount,urlchaptercount)
           else:
               print "Do update - epub(%d) vs url(%d)" % (chaptercount, urlchaptercount)
-               ## Get updated title page/metadata by itself in an epub.
-               ## Even if the title page isn't included, this carries the metadata.
-               titleio = StringIO()
-               writeStory(config,adapter,"epub",metaonly=True,outstream=titleio)
-
-               newchaptersio = None
               if not options.metaonly:
-                   ## Go get the new chapters only in another epub.
-                   newchaptersio = StringIO()
-                   adapter.setChaptersRange(chaptercount+1,urlchaptercount)
-                   config.set("overrides",'include_tocpage','false')
-                   config.set("overrides",'include_titlepage','false')
-                   writeStory(config,adapter,"epub",outstream=newchaptersio)
-               
-               # out = open("testing/titleio.epub","wb")
-               # out.write(titleio.getvalue())
-               # out.close()
-               
-               # out = open("testing/updateio.epub","wb")
-               # out.write(updateio.getvalue())
-               # out.close()
-               
-               # out = open("testing/newchaptersio.epub","wb")
-               # out.write(newchaptersio.getvalue())
-               # out.close()
-               
-               ## Merge the three epubs together.
-               doMerge(args[0],
-                       [titleio,updateio,newchaptersio],
-                       fromfirst=True,
-                       titlenavpoints=False,
-                       striptitletoc=False,
-                       forceunique=False)

+                   # update now handled by pre-populating the old
+                   # images and chapters in the adapter rather than
+                   # merging epubs.
+                   (url,chaptercount,
+                    adapter.oldchapters,
+                    adapter.oldimgs) = get_update_data(args[0])
+
+                   writeStory(config,adapter,"epub")
+                   
       else:
           # regular download
           if options.metaonly:
@@ -16,374 +16,10 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-import sys
-import os
-import re
-#import StringIO
-from optparse import OptionParser      
-
-import zlib
-import zipfile
-from zipfile import ZipFile, ZIP_STORED, ZIP_DEFLATED
-from time import time
-
-from exceptions import KeyError
-
-from xml.dom.minidom import parse, parseString, getDOMImplementation
-        
-def main(argv):
-    # read in args, anything starting with -- will be treated as --<varible>=<value>
-    usage = "usage: %prog [options] <input epub> [<input epub>...]"
-    parser = OptionParser(usage)
-    parser.add_option("-o", "--output", dest="outputopt", default="merge.epub",
-                      help="Set OUTPUT file, Default: merge.epub", metavar="OUTPUT")
-    parser.add_option("-t", "--title", dest="titleopt", default=None,
-                      help="Use TITLE as the metadata title.  Default: '<first epub title> Anthology'", metavar="TITLE")
-    parser.add_option("-d", "--description", dest="descopt", default=None,
-                      help="Use DESC as the metadata description.  Default: '<epub title> by <author>' for each epub.", metavar="DESC")
-    parser.add_option("-a", "--author",
-                      action="append", dest="authoropts", default=[],
-                      help="Use AUTHOR as a metadata author, multiple authors may be given, Default: <All authors from epubs>", metavar="AUTHOR")
-    parser.add_option("-f", "--first",
-                      action="store_true", dest="fromfirst", default=False,
-                      help="Take all metadata from first input epub",)
-    parser.add_option("-n", "--titles-in-toc",
-                      action="store_true", dest="titlenavpoints",
-                      help="Put an entry in the TOC for each epub, in addition to each epub's chapters.",)
-    parser.add_option("-s", "--strip-title-toc",
-                      action="store_true", dest="striptitletoc",
-                      help="Strip any title_page.xhtml and toc_page.xhtml files.",)
-    
-    (options, args) = parser.parse_args()
-
-    ## Add .epub if not already there.
-    if not options.outputopt.lower().endswith(".epub"):
-        options.outputopt=options.outputopt+".epub"
-
-    print "output file: "+options.outputopt
-    doMerge(options.outputopt,
-            args,
-            options.authoropts,
-            options.titleopt,
-            options.descopt,
-            options.fromfirst,
-            options.titlenavpoints,
-            options.striptitletoc)
-
-    # output = StringIO.StringIO()
-    # files = []
-    # for file in args:
-    #     f = open(file,"rb")
-    #     fio = StringIO.StringIO(f.read())
-    #     f.close()
-    #     files.append(fio)
-
-    # doMerge(output,files,authoropts,titleopt,descopt,fromfirst,titlenavpoints,striptitletoc)
-
-    # out = open(outputopt,"wb")
-    # out.write(output.getvalue())
-    
-def doMerge(outputio,files,authoropts=[],titleopt=None,descopt=None,
-            fromfirst=False,
-            titlenavpoints=True,
-            striptitletoc=False,
-            forceunique=True):
-    '''
-    outputio = output file name or StringIO.
-    files = list of input file names or StringIOs.
-    authoropts = list of authors to use, otherwise add from all input
-    titleopt = title, otherwise '<first title> Anthology'
-    descopt = description, otherwise '<title> by <author>' list for all input
-    fromfirst if true, take all metadata (including author, title, desc) from first input
-    titlenavpoints if true, put in a new TOC entry for each epub
-    striptitletoc if true, strip out any (title|toc)_page.xhtml files
-    forceunique if true, guarantee uniqueness of contents by adding a dir for each input
-    '''
-    ## Python 2.5 ZipFile is rather more primative than later
-    ## versions.  It can operate on a file, or on a StringIO, but
-    ## not on an open stream.  OTOH, I suspect we would have had
-    ## problems with closing and opening again to change the
-    ## compression type anyway.
-
-    filecount=0
-    source=None
-    
-    ## Write mimetype file, must be first and uncompressed.
-    ## Older versions of python(2.4/5) don't allow you to specify
-    ## compression by individual file.
-    ## Overwrite if existing output file.
-    outputepub = ZipFile(outputio, "w", compression=ZIP_STORED)
-    outputepub.debug = 3
-    outputepub.writestr("mimetype", "application/epub+zip")
-    outputepub.close()
-
-    ## Re-open file for content.
-    outputepub = ZipFile(outputio, "a", compression=ZIP_DEFLATED)
-    outputepub.debug = 3
-
-    ## Create META-INF/container.xml file.  The only thing it does is
-    ## point to content.opf
-    containerdom = getDOMImplementation().createDocument(None, "container", None)
-    containertop = containerdom.documentElement
-    containertop.setAttribute("version","1.0")
-    containertop.setAttribute("xmlns","urn:oasis:names:tc:opendocument:xmlns:container")
-    rootfiles = containerdom.createElement("rootfiles")
-    containertop.appendChild(rootfiles)
-    rootfiles.appendChild(newTag(containerdom,"rootfile",{"full-path":"content.opf",
-                                                          "media-type":"application/oebps-package+xml"}))
-    outputepub.writestr("META-INF/container.xml",containerdom.toprettyxml(indent='   ',encoding='utf-8'))    
-
-    ## Process input epubs.
-    
-    items = [] # list of (id, href, type) tuples(all strings) -- From .opfs' manifests
-    items.append(("ncx","toc.ncx","application/x-dtbncx+xml")) ## we'll generate the toc.ncx file,
-                                                               ## but it needs to be in the items manifest.
-    itemrefs = [] # list of strings -- idrefs from .opfs' spines
-    navmaps = [] # list of navMap DOM elements -- TOC data for each from toc.ncx files
-
-    booktitles = [] # list of strings -- Each book's title
-    allauthors = [] # list of lists of strings -- Each book's list of authors.
-
-    filelist = []
-    
-    booknum=1
-    firstmetadom = None
-    for file in files:
-        if file == None : continue
-        
-        book = "%d" % booknum
-        bookdir = ""
-        bookid = ""
-        if forceunique:
-            bookdir = "%d/" % booknum
-            bookid = "a%d" % booknum
-        #print "book %d" % booknum
-        
-        epub = ZipFile(file, 'r')
-
-        ## Find the .opf file.
-        container = epub.read("META-INF/container.xml")
-        containerdom = parseString(container)
-        rootfilenodelist = containerdom.getElementsByTagName("rootfile")
-        rootfilename = rootfilenodelist[0].getAttribute("full-path")
-
-        ## Save the path to the .opf file--hrefs inside it are relative to it.
-        relpath = os.path.dirname(rootfilename)
-        if( len(relpath) > 0 ):
-            relpath=relpath+"/"
-            
-        metadom = parseString(epub.read(rootfilename))
-        if booknum==1:
-            firstmetadom = metadom.getElementsByTagName("metadata")[0]
-            try:
-                source=firstmetadom.getElementsByTagName("dc:source")[0].firstChild.data.encode("utf-8")
-            except:
-                source=""
-            #print "Source:%s"%source
-
-        ## Save indiv book title
-        booktitles.append(metadom.getElementsByTagName("dc:title")[0].firstChild.data)
-
-        ## Save authors.
-        authors=[]
-        for creator in metadom.getElementsByTagName("dc:creator"):
-            if( creator.getAttribute("opf:role") == "aut" ):
-                authors.append(creator.firstChild.data)
-        allauthors.append(authors)
-
-        for item in metadom.getElementsByTagName("item"):
-            if( item.getAttribute("media-type") == "application/x-dtbncx+xml" ):
-                # TOC file is only one with this type--as far as I know.
-                # grab the whole navmap, deal with it later.
-                tocdom = parseString(epub.read(relpath+item.getAttribute("href")))
-                
-                for navpoint in tocdom.getElementsByTagName("navPoint"):
-                    navpoint.setAttribute("id",bookid+navpoint.getAttribute("id"))
-
-                for content in tocdom.getElementsByTagName("content"):
-                    content.setAttribute("src",bookdir+relpath+content.getAttribute("src"))
-
-                navmaps.append(tocdom.getElementsByTagName("navMap")[0])
-            else:
-                id=bookid+item.getAttribute("id")
-                href=bookdir+relpath+item.getAttribute("href")
-                href=href.encode('utf8')
-                #print "href:"+href
-                if not striptitletoc or not re.match(r'.*/(title|toc)_page\.xhtml',
-                                                      item.getAttribute("href")):
-                    if href not in filelist:
-                        try:
-                            outputepub.writestr(href,
-                                                epub.read(relpath+item.getAttribute("href")))
-                            if re.match(r'.*/(file|chapter)\d+\.x?html',href):
-                                filecount+=1
-                            items.append((id,href,item.getAttribute("media-type")))
-                            filelist.append(href)
-                        except KeyError, ke:
-                            pass # Skip missing files.
-                
-        for itemref in metadom.getElementsByTagName("itemref"):
-
-            if not striptitletoc or not re.match(r'(title|toc)_page', itemref.getAttribute("idref")):
-                itemrefs.append(bookid+itemref.getAttribute("idref"))
-
-        booknum=booknum+1;
-        if not forceunique:
-            # If not forceunique, it's an epub update.
-            # If there's a "calibre_bookmarks.txt", it's from reading
-            # in Calibre and should be preserved.
-            try:
-                fn = "META-INF/calibre_bookmarks.txt"
-                outputepub.writestr(fn,epub.read(fn))
-            except:
-                pass
-        
-
-    ## create content.opf file. 
-    uniqueid="epubmerge-uid-%d" % time() # real sophisticated uid scheme.
-    contentdom = getDOMImplementation().createDocument(None, "package", None)
-    package = contentdom.documentElement
-    if fromfirst and firstmetadom:
-        metadata = firstmetadom
-        firstpackage = firstmetadom.parentNode
-        package.setAttribute("version",firstpackage.getAttribute("version"))
-        package.setAttribute("xmlns",firstpackage.getAttribute("xmlns"))
-        package.setAttribute("unique-identifier",firstpackage.getAttribute("unique-identifier"))
-    else:
-        package.setAttribute("version","2.0")
-        package.setAttribute("xmlns","http://www.idpf.org/2007/opf")
-        package.setAttribute("unique-identifier","epubmerge-id")
-        metadata=newTag(contentdom,"metadata",
-                        attrs={"xmlns:dc":"http://purl.org/dc/elements/1.1/",
-                               "xmlns:opf":"http://www.idpf.org/2007/opf"})
-        metadata.appendChild(newTag(contentdom,"dc:identifier",text=uniqueid,attrs={"id":"epubmerge-id"}))
-        if( titleopt is None ):
-            titleopt = booktitles[0]+" Anthology"
-        metadata.appendChild(newTag(contentdom,"dc:title",text=titleopt))
-    
-        # If cmdline authors, use those instead of those collected from the epubs
-        # (allauthors kept for TOC & description gen below.
-        if( len(authoropts) > 1  ):
-            useauthors=[authoropts]
-        else:
-            useauthors=allauthors
-            
-        usedauthors=dict()
-        for authorlist in useauthors:
-            for author in authorlist:
-                if( not usedauthors.has_key(author) ):
-                    usedauthors[author]=author
-                    metadata.appendChild(newTag(contentdom,"dc:creator",
-                                                attrs={"opf:role":"aut"},
-                                                text=author))
-    
-        metadata.appendChild(newTag(contentdom,"dc:contributor",text="epubmerge",attrs={"opf:role":"bkp"}))
-        metadata.appendChild(newTag(contentdom,"dc:rights",text="Copyrights as per source stories"))
-        metadata.appendChild(newTag(contentdom,"dc:language",text="en"))
-    
-        if not descopt:
-            # created now, but not filled in until TOC generation to save loops.
-            description = newTag(contentdom,"dc:description",text="Anthology containing:\n")
-        else:
-            description = newTag(contentdom,"dc:description",text=descopt)
-        metadata.appendChild(description)
-    
-    package.appendChild(metadata)
-    
-    manifest = contentdom.createElement("manifest")
-    package.appendChild(manifest)
-    for item in items:
-        (id,href,type)=item
-        manifest.appendChild(newTag(contentdom,"item",
-                                       attrs={'id':id,
-                                              'href':href,
-                                              'media-type':type}))
-        
-    spine = newTag(contentdom,"spine",attrs={"toc":"ncx"})
-    package.appendChild(spine)
-    for itemref in itemrefs:
-        spine.appendChild(newTag(contentdom,"itemref",
-                                    attrs={"idref":itemref,
-                                           "linear":"yes"}))
-
-    ## create toc.ncx file
-    tocncxdom = getDOMImplementation().createDocument(None, "ncx", None)
-    ncx = tocncxdom.documentElement
-    ncx.setAttribute("version","2005-1")
-    ncx.setAttribute("xmlns","http://www.daisy.org/z3986/2005/ncx/")
-    head = tocncxdom.createElement("head")
-    ncx.appendChild(head)
-    head.appendChild(newTag(tocncxdom,"meta",
-                            attrs={"name":"dtb:uid", "content":uniqueid}))
-    head.appendChild(newTag(tocncxdom,"meta",
-                            attrs={"name":"dtb:depth", "content":"1"}))
-    head.appendChild(newTag(tocncxdom,"meta",
-                            attrs={"name":"dtb:totalPageCount", "content":"0"}))
-    head.appendChild(newTag(tocncxdom,"meta",
-                            attrs={"name":"dtb:maxPageNumber", "content":"0"}))
-    
-    docTitle = tocncxdom.createElement("docTitle")
-    docTitle.appendChild(newTag(tocncxdom,"text",text=titleopt))
-    ncx.appendChild(docTitle)
-    
-    tocnavMap = tocncxdom.createElement("navMap")
-    ncx.appendChild(tocnavMap)
-
-    ## TOC navPoints can be nested, but this flattens them for
-    ## simplicity, plus adds a navPoint for each epub.
-    booknum=0
-    for navmap in navmaps:
-        navpoints = navmap.getElementsByTagName("navPoint")
-        if titlenavpoints:
-           ## Copy first navPoint of each epub, give a different id and
-           ## text: bookname by authorname
-            newnav = navpoints[0].cloneNode(True)
-            newnav.setAttribute("id","book"+newnav.getAttribute("id"))
-            ## For purposes of TOC titling & desc, use first book author
-            newtext = newTag(tocncxdom,"text",text=booktitles[booknum]+" by "+allauthors[booknum][0])            
-            text = newnav.getElementsByTagName("text")[0]
-            text.parentNode.replaceChild(newtext,text)
-            tocnavMap.appendChild(newnav)
-        
-        if not descopt and not fromfirst:
-            description.appendChild(contentdom.createTextNode(booktitles[booknum]+" by "+allauthors[booknum][0]+"\n"))
-            
-        for navpoint in navpoints:
-            #print "navpoint:%s"%navpoint.getAttribute("id")
-            if not striptitletoc or not re.match(r'(title|toc)_page',navpoint.getAttribute("id")):
-                tocnavMap.appendChild(navpoint)
-        booknum=booknum+1;
-
-    ## Force strict ordering of playOrder
-    playorder=1
-    for navpoint in tocncxdom.getElementsByTagName("navPoint"):
-        navpoint.setAttribute("playOrder","%d" % playorder)
-        if( not navpoint.getAttribute("id").startswith("book") ):
-            playorder = playorder + 1
-
-    ## content.opf written now due to description being filled in
-    ## during TOC generation to save loops.
-    outputepub.writestr("content.opf",contentdom.toxml('utf-8'))
-    outputepub.writestr("toc.ncx",tocncxdom.toxml('utf-8'))
-
-    # declares all the files created by Windows.  otherwise, when
-    # it runs in appengine, windows unzips the files as 000 perms.
-    for zf in outputepub.filelist:
-        zf.create_system = 0
-    outputepub.close()
-
-    return (source,filecount)
-
-## Utility method for creating new tags.
-def newTag(dom,name,attrs=None,text=None):
-    tag = dom.createElement(name)
-    if( attrs is not None ):
-        for attr in attrs.keys():
-            tag.setAttribute(attr,attrs[attr])
-    if( text is not None ):
-        tag.appendChild(dom.createTextNode(text))
-    return tag
    
 if __name__ == "__main__":
-    main(sys.argv[1:])
+    print('''
+The this utility has been split out into it's own project. 
+See: http://code.google.com/p/epubmerge/
+...for a CLI epubmerge.py program and calibre plugin.
+''')
@@ -27,7 +27,6 @@ from .. import exceptions as exceptions
 import adapter_test1
 import adapter_fanfictionnet
 import adapter_castlefansorg
-import adapter_fanfictionnet
 import adapter_fictionalleyorg
 import adapter_fictionpresscom
 import adapter_ficwadcom
@@ -45,6 +44,17 @@ import adapter_whoficcom
 import adapter_siyecouk
 import adapter_archiveofourownorg
 import adapter_ficbooknet
+import adapter_portkeyorg
+import adapter_mugglenetcom
+import adapter_hpfandomnet
+import adapter_thequidditchpitchorg
+import adapter_nfacommunitycom
+import adapter_midnightwhispersca
+import adapter_ksarchivecom
+import adapter_archiveskyehawkecom
+import adapter_squidgeorgpeja
+import adapter_libraryofmoriacom
+import adapter_wraithbaitcom

 ## This bit of complexity allows adapters to be added by just adding
 ## importing.  It eliminates the long if/else clauses we used to need
@@ -63,7 +73,7 @@ for x in imports():
        #print x
        __class_list.append(sys.modules[x].getClass())
            
-def getAdapter(config,url):
+def getAdapter(config,url,fileform=None):
    ## fix up leading protocol.
    fixedurl = re.sub(r"(?i)^[htp]+[:/]+","http://",url.strip())
    if not fixedurl.startswith("http"):
@@ -88,10 +98,14 @@ def getAdapter(config,url):
        fixedurl = fixedurl.replace("http://","http://www.")
    if cls:
        adapter = cls(config,fixedurl) # raises InvalidStoryURL
+        adapter.setSectionOrder(adapter.getConfigSection(),fileform)
        return adapter
    # No adapter found.
    raise exceptions.UnknownSite( url, [cls.getSiteDomain() for cls in __class_list] )

+def getConfigSections():
+    return [cls.getConfigSection() for cls in __class_list]
+
 def getClassFor(domain):
    for cls in __class_list:
        if cls.matchesSite(domain):
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class AdAstraFanficComSiteAdapter(BaseSiteAdapter):

@@ -133,7 +133,8 @@ class AdAstraFanficComSiteAdapter(BaseSiteAdapter):
                # sometimes poorly formated desc (<p> w/o </p>) leads
                # to all labels being included.
                svalue=svalue[:svalue.find('<span class="label">')]
-                self.story.setMetadata('description',stripHTML(svalue))
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))

            if 'Rated' in label:
                self.story.setMetadata('rating', value)
@@ -220,7 +221,7 @@ class AdAstraFanficComSiteAdapter(BaseSiteAdapter):
        if None == span:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(span)
+        return self.utf8FromSoup(url,span)

 def getClass():
    return AdAstraFanficComSiteAdapter
@@ -24,7 +24,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 def getClass():
    return ArchiveOfOurOwnOrgAdapter
@@ -101,7 +101,7 @@ class ArchiveOfOurOwnOrgAdapter(BaseSiteAdapter):
        # Now go hunting for all the meta data and the chapter list.
        
        ## Title
-        a = soup.find('a', href=re.compile(r"^/works/\w+"))
+        a = soup.find('a', href=re.compile(r"^/works/\d+$"))
        self.story.setMetadata('title',a.string)
 		
        # Find authorid and URL from... author url.
@@ -126,7 +126,8 @@ class ArchiveOfOurOwnOrgAdapter(BaseSiteAdapter):

        a = metasoup.find('blockquote',{'class':'userstuff'})
        if a != None:
-            self.story.setMetadata('description',a.text)
+            self.setDescription(url,a.text)
+            #self.story.setMetadata('description',a.text)
 		
        a = metasoup.find('dd',{'class':"rating tags"})
        if a != None:
@@ -134,15 +135,13 @@ class ArchiveOfOurOwnOrgAdapter(BaseSiteAdapter):
 		
        a = metasoup.find('dd',{'class':"fandom tags"})
        fandoms = a.findAll('a',{'class':"tag"})
-        fandomstext = [fandom.string for fandom in fandoms]
-        for fandom in fandomstext:
+        for fandom in fandoms:
            self.story.addToList('category',fandom.string)
 		
        a = metasoup.find('dd',{'class':"warning tags"})
        if a != None:
            warnings = a.findAll('a',{'class':"tag"})
-            warningstext = [warning.string for warning in warnings]
-            for warning in warningstext:
+            for warning in warnings:
                if warning.string == "Author Chose Not To Use Archive Warnings":
                    warning.string = "No Archive Warnings Apply"
                if warning.string != "No Archive Warnings Apply":
@@ -151,28 +150,24 @@ class ArchiveOfOurOwnOrgAdapter(BaseSiteAdapter):
        a = metasoup.find('dd',{'class':"freeform tags"})
        if a != None:
            genres = a.findAll('a',{'class':"tag"})
-            genrestext = [genre.string for genre in genres]
-            for genre in genrestext:
+            for genre in genres:
                self.story.addToList('genre',genre.string)
        a = metasoup.find('dd',{'class':"category tags"})
        if a != None:
            genres = a.findAll('a',{'class':"tag"})
-            genrestext = [genre.string for genre in genres]
-            for genre in genrestext:
+            for genre in genres:
                if genre != "Gen":
                    self.story.addToList('genre',genre.string)
 		
        a = metasoup.find('dd',{'class':"character tags"})
        if a != None:
            chars = a.findAll('a',{'class':"tag"})
-            charstext = [char.string for char in chars]
-            for char in charstext:
+            for char in chars:
                self.story.addToList('characters',char.string)
        a = metasoup.find('dd',{'class':"relationship tags"})
        if a != None:
            chars = a.findAll('a',{'class':"tag"})
-            charstext = [char.string for char in chars]
-            for char in charstext:
+            for char in chars:
                self.story.addToList('characters',char.string)
 		

@@ -222,25 +217,26 @@ class ArchiveOfOurOwnOrgAdapter(BaseSiteAdapter):
        logging.debug('Getting chapter text from: %s' % url)
 		
        chapter=bs.BeautifulSoup('<div class="story"></div>')
-        soup = bs.BeautifulSoup(self._fetchUrl(url),selfClosingTags=('br','hr'))
+        data = self._fetchUrl(url)
+        soup = bs.BeautifulSoup(data,selfClosingTags=('br','hr'))
 		
        headnotes = soup.find('div', {'class' : "preface group"}).find('div', {'class' : "notes module"})
        if headnotes != None:
            headnotes = headnotes.find('blockquote', {'class' : "userstuff"})
            if headnotes != None:
-                chapter.append(bs.BeautifulSoup("<b>Author's Note:</b>"))
+                chapter.append("<b>Author's Note:</b>")
                chapter.append(headnotes)
        
        chapsumm = soup.find('div', {'id' : "summary"})
        if chapsumm != None:
            chapsumm = chapsumm.find('blockquote')
-            chapter.append(bs.BeautifulSoup("<b>Summary for the Chapter:</b>"))
+            chapter.append("<b>Summary for the Chapter:</b>")
            chapter.append(chapsumm)
        chapnotes = soup.find('div', {'id' : "notes"})
        if chapnotes != None:
            chapnotes = chapnotes.find('blockquote')
            if chapnotes != None:
-                chapter.append(bs.BeautifulSoup("<b>Notes for the Chapter:</b>"))
+                chapter.append("<b>Notes for the Chapter:</b>")
                chapter.append(chapnotes)
 		
        text = soup.find('div', {'class' : "userstuff module"})
@@ -252,16 +248,16 @@ class ArchiveOfOurOwnOrgAdapter(BaseSiteAdapter):
        chapfoot = soup.find('div', {'class' : "end notes module", 'role' : "complementary"})
        if chapfoot != None:
            chapfoot = chapfoot.find('blockquote')
-            chapter.append(bs.BeautifulSoup("<b>Notes for the Chapter:</b>"))
+            chapter.append("<b>Notes for the Chapter:</b>")
            chapter.append(chapfoot)
 		
        footnotes = soup.find('div', {'id' : "work_endnotes"})
        if footnotes != None:
            footnotes = footnotes.find('blockquote')
-            chapter.append(bs.BeautifulSoup("<b>Author's Note:</b>"))
+            chapter.append("<b>Author's Note:</b>")
            chapter.append(footnotes)
 			
        if None == soup:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
-    
-        return utf8FromSoup(chapter)
+
+        return self.utf8FromSoup(url,chapter)
@@ -0,0 +1,190 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+
+def getClass():
+    return ArchiveSkyeHawkeComAdapter
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class ArchiveSkyeHawkeComAdapter(BaseSiteAdapter):
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
+        self.password = ""
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        self._setURL('http://' + self.getSiteDomain() + '/story.php?no='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','ash')
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%Y-%m-%d"
+            
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'archive.skyehawke.com'
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/story.php?no=1234"
+
+    def getSiteURLPattern(self):
+        return re.escape("http://"+self.getSiteDomain()+"/story.php?no=")+r"\d+$"
+
+
+        
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        url = self.url
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        data = self._fetchUrl(url)
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        # print data
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        ## Title
+        a = soup.find('div', {'class':"story border"}).find('span',{'class':'left'})
+        title=a.text.split('"')[1]
+        self.story.setMetadata('title',title)
+        
+        # Find authorid and URL from... author url.
+        author = a.find('a')
+        self.story.setMetadata('authorId',author['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/'+author['href'])
+        self.story.setMetadata('author',author.string)
+		
+        authorSoup = bs.BeautifulSoup(self._fetchUrl(self.story.getMetadata('authorUrl')))
+		
+        chapter=soup.find('select',{'name':'chapter'}).findAll('option')
+	
+        for i in range(1,len(chapter)):
+            ch=chapter[i]
+            self.chapterUrls.append((stripHTML(ch),ch['value']))
+		
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+		
+        box=soup.find('div', {'class': "container borderridge"})
+        sum=box.find('span').text
+        self.setDescription(url,sum)
+
+        boxes=soup.findAll('div', {'class': "container bordersolid"})
+        for box in boxes:
+            if box.find('b') != None and box.find('b').text == "History and Story Information":
+			
+                for b in box.findAll('b'):
+                    if "words" in b.nextSibling:
+                        self.story.setMetadata('numWords', b.text)
+                    if "archived" in b.previousSibling:
+                        self.story.setMetadata('datePublished', makeDate(stripHTML(b.text), self.dateformat))
+                    if "updated" in b.previousSibling:
+                        self.story.setMetadata('dateUpdated', makeDate(stripHTML(b.text), self.dateformat))
+                    if "fandom" in b.nextSibling:
+                        self.story.addToList('category', b.text)
+						
+                for br in box.findAll('br'):
+                    br.replaceWith('split')
+                genre=box.text.split("Genre:")[1].split("split")[0]
+                if not "Unspecified" in genre:
+                    self.story.addToList('genre',genre)
+			
+			
+            if box.find('span') != None and box.find('span').text == "WARNING":
+			
+                rating=box.findAll('span')[1]
+                rating.find('br').replaceWith('split')
+                rating=rating.text.replace("This story is rated",'').split('split')[0]
+                self.story.setMetadata('rating',rating)
+                logging.debug(self.story.getMetadata('rating'))
+			
+                warnings=box.find('ol')
+                if warnings != None:
+                    warnings=warnings.text.replace(']', '').replace('[', '').split('  ')
+                    for warning in warnings:
+                        self.story.addToList('warnings',warning)
+	
+	
+        for asoup in authorSoup.findAll('div', {'class':"story bordersolid"}):
+            if asoup.find('a')['href'] == 'story.php?no='+self.story.getMetadata('storyId'):
+                if '[ Completed ]' in asoup.text:
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+                chars=asoup.findNext('div').text.split('Characters')[1].split(']')[0]
+                for char in chars.split(','):
+                    if not "None" in char:
+                        self.story.addToList('characters',char)
+                break
+	
+	
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        soup = bs.BeautifulStoneSoup(self._fetchUrl(url),
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+        
+        div = soup.find('div',{'class':"chapter bordersolid"}).findNext('div').findNext('div')
+
+        if None == div:
+            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,div)
@@ -24,7 +24,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 # By virtue of being recent and requiring both is_adult and user/pass,
 # adapter_fanficcastletvnet.py is the best choice for learning to
@@ -74,6 +74,7 @@ class CastleFansOrgAdapter(BaseSiteAdapter): # XXX
        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
        
        # normalized story URL.
+        # XXX Most sites don't have the /fanfic part.  Replace all to remove it usually.
        self._setURL('http://' + self.getSiteDomain() + '/fanfic/viewstory.php?sid='+self.story.getMetadata('storyId'))
        
        # Each adapter needs to have a unique site abbreviation.
@@ -218,7 +219,8 @@ class CastleFansOrgAdapter(BaseSiteAdapter): # XXX
                while not defaultGetattr(value,'class') == 'label':
                    svalue += str(value)
                    value = value.nextSibling
-                self.story.setMetadata('description',stripHTML(svalue))
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))

            if 'Rated' in label:
                self.story.setMetadata('rating', value)
@@ -305,4 +307,4 @@ class CastleFansOrgAdapter(BaseSiteAdapter): # XXX
        if None == div:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(div)
+        return self.utf8FromSoup(url,div)
@@ -24,7 +24,7 @@ import time
 from .. import BeautifulSoup as bs
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class FanFictionNetSiteAdapter(BaseSiteAdapter):

@@ -74,6 +74,7 @@ class FanFictionNetSiteAdapter(BaseSiteAdapter):
        # use BeautifulSoup HTML parser to make everything easier to find.
        try:
            data = self._fetchUrl(url)
+            #print("\n===================\n%s\n===================\n"%data)
            soup = bs.BeautifulSoup(data)
        except urllib2.HTTPError, e:
            if e.code == 404:
@@ -141,7 +142,7 @@ class FanFictionNetSiteAdapter(BaseSiteAdapter):
                continue
            if 'var storyid' in script.string:
                for line in script.string.split('\n'):
-                    m = re.match(r"^ +var ([^ ]+) = '?(.*?)'?;$",line)
+                    m = re.match(r"^ +var ([^ ]+) = '?(.*?)'?;\r?$",line)
                    if m == None : continue
                    var,value = m.groups()
                    # remove javascript escaping from values.
@@ -152,7 +153,8 @@ class FanFictionNetSiteAdapter(BaseSiteAdapter):
                    if 'title_t' in var:
                        self.story.setMetadata('title', value)
                    if 'summary' in var:
-                        self.story.setMetadata('description', value)
+                        self.setDescription(url,value)
+                        #self.story.setMetadata('description', value)
                    if 'datep' in var:
                        self.story.setMetadata('datePublished',makeDate(value, '%m-%d-%y'))
                    if 'dateu' in var:
@@ -269,7 +271,7 @@ class FanFictionNetSiteAdapter(BaseSiteAdapter):
            logging.debug('div id=storytext not found.  data:%s'%data)
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)

-        return utf8FromSoup(div)
+        return self.utf8FromSoup(url,div)

 def getClass():
    return FanFictionNetSiteAdapter
@@ -26,7 +26,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate


 def getClass():
@@ -201,7 +201,8 @@ class FicBookNetAdapter(BaseSiteAdapter):
                break
 				
        summary=soup.find('span', {'class' : 'urlize'})
-        self.story.setMetadata('description', summary.text)
+        self.setDescription(url,summary.text)
+        #self.story.setMetadata('description', summary.text)
            
    # grab the text for an individual chapter.
    def getChapterText(self, url):
@@ -218,4 +219,4 @@ class FicBookNetAdapter(BaseSiteAdapter):
        if None == chapter:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(chapter)
+        return self.utf8FromSoup(url,chapter)
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class FictionAlleyOrgSiteAdapter(BaseSiteAdapter):

@@ -187,7 +187,8 @@ class FictionAlleyOrgSiteAdapter(BaseSiteAdapter):
            
        for small in storydd.findAll('small'):
            small.extract() ## removes the <small> tags, leaving only the summary.
-        self.story.setMetadata('description',stripHTML(storydd))
+        self.setDescription(url,storydd)
+        #self.story.setMetadata('description',stripHTML(storydd))
        
        return

@@ -223,7 +224,7 @@ class FictionAlleyOrgSiteAdapter(BaseSiteAdapter):
        if not data or not text:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(text)
+        return self.utf8FromSoup(url,text)

 def getClass():
    return FictionAlleyOrgSiteAdapter
@@ -26,7 +26,7 @@ from .. import BeautifulSoup as bs
 from .. import exceptions as exceptions
 from ..htmlcleanup import stripHTML

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class FicwadComSiteAdapter(BaseSiteAdapter):

@@ -124,7 +124,8 @@ class FicwadComSiteAdapter(BaseSiteAdapter):

        # description
        storydiv = soup.find("div",{"id":"story"})
-        self.story.setMetadata('description', storydiv.find("blockquote",{'class':'summary'}).p.string)
+        self.setDescription(url,storydiv.find("blockquote",{'class':'summary'}).p.string)
+        #self.story.setMetadata('description', storydiv.find("blockquote",{'class':'summary'}).p.string)

        # most of the meta data is here:
        metap = storydiv.find("p",{"class":"meta"})
@@ -209,7 +210,7 @@ class FicwadComSiteAdapter(BaseSiteAdapter):
        if None == span:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)

-        return utf8FromSoup(span)
+        return self.utf8FromSoup(url,span)

 def getClass():
    return FicwadComSiteAdapter
@@ -20,13 +20,14 @@ import logging
 import re
 import urllib2
 import cookielib as cl
-import datetime
+from datetime import datetime
+import json

 from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 def getClass():
    return FimFictionNetSiteAdapter
@@ -36,7 +37,7 @@ class FimFictionNetSiteAdapter(BaseSiteAdapter):
    def __init__(self, config, url):
        BaseSiteAdapter.__init__(self, config, url)
        self.story.setMetadata('siteabbrev','fimficnet')
-        self.story.setMetadata('storyId',self.parsedUrl.path.split('/',)[2])
+        self.story.setMetadata('storyId', self.parsedUrl.path.split('/',)[2])
        self._setURL("http://"+self.getSiteDomain()+"/story/"+self.story.getMetadata('storyId')+"/")
        self.is_adult = False
        
@@ -74,6 +75,10 @@ class FimFictionNetSiteAdapter(BaseSiteAdapter):
            self.opener = urllib2.build_opener(cookieproc)
        
        try:
+            apiResponse = urllib2.urlopen("http://www.fimfiction.net/api/story.php?story=%s" % (self.story.getMetadata("storyId"))).read()
+            apiData = json.loads(apiResponse)
+            
+            # Unfortunately, we still need to load the story index page to parse the characters
            data = self._fetchUrl(self.url)
        except urllib2.HTTPError, e:
            if e.code == 404:
@@ -83,93 +88,81 @@ class FimFictionNetSiteAdapter(BaseSiteAdapter):
        
        if "Warning: mysql_fetch_array(): supplied argument is not a valid MySQL result resource" in data:
            raise exceptions.StoryDoesNotExist(self.url)
+
+        if "/images/missing_story.png" in data:
+            raise exceptions.StoryDoesNotExist(self.url)
+        
+        if "Invalid story id" in apiData.values():
+            raise exceptions.StoryDoesNotExist(self.url)
        
        if "This story has been marked as having adult content." in data:
            raise exceptions.AdultCheckRequired(self.url)
        
-        soup = bs.BeautifulSoup(data).find("div", {"class":"content_box post_content_box"})
+        if self.password:
+            params = {}
+            params['password'] = self.password
+            data = self._postUrl(self.url,params)

-        titleheader = soup.find("h2")
-        title = titleheader.find("a", href=re.compile(r'^/story/')).text
-        author = titleheader.find("a", href=re.compile(r'^/user/')).text
-        self.story.setMetadata("title", title)
-        self.story.setMetadata("author", author)
-        self.story.setMetadata("authorId", author) # The author's name will be unique
-        self.story.setMetadata("authorUrl", "http://%s/user/%s" % (self.getSiteDomain(),author))
+        if "Enter the password the author set for this story to view it." in data:
+            if self.getConfig('fail_on_password'):
+                raise exceptions.FailedToDownload("%s requires story password and fail_on_password is true."%self.url)
+            else:
+                raise exceptions.FailedToLogin(self.url,"Story requires individual password",passwdonly=True)
+         
+        storyMetadata = apiData["story"]    
+            
+        self.story.setMetadata("title", storyMetadata["title"])
+        self.story.setMetadata("author", storyMetadata["author"]["name"])
+        self.story.setMetadata("authorId", storyMetadata["author"]["id"])
+        self.story.setMetadata("authorUrl", "http://%s/user/%s" % (self.getSiteDomain(), storyMetadata["author"]["name"]))
        
-        chapterDates = []
-
-        for chapter in soup.findAll("a", {"class":"chapter_link"}):
-            chapterDates.append(chapter.span.extract().text.strip("()"))
-            self.chapterUrls.append((chapter.text.strip(), "http://"+self.getSiteDomain() + chapter['href']))
+        chapters = [{"chapterTitle": chapter["title"], "chapterURL": chapter["link"]} for chapter in storyMetadata["chapters"]]
+        for chapter in chapters:
+            self.chapterUrls.append((chapter["chapterTitle"], chapter["chapterURL"]))
+        self.story.setMetadata("numChapters", len(self.chapterUrls))
        
-        self.story.setMetadata('numChapters',len(self.chapterUrls))
-        
-        for character in [character_icon['title'] for character_icon in soup.findAll("a", {"class":"character_icon"})]:
-            self.story.addToList("characters", character)
-        for category in [category.text for category in soup.find("div", {"class":"categories"}).findAll("a")]:
-            self.story.addToList("genre", category)
-        self.story.addToList("category", "My Little Pony")
-        
-        
-        # The very last list element in the list of chapters contains the status, rating and word count e.g.:
-        #
-        #    <li>
-        #       Incomplete | Rating:
-        #       <span style="color:#c78238;">Teen</span>
-        #       <div class="word_count"><b>5,203</b>words total</div>
-        #    </li>
-        #
-
-        status_bar = soup.findAll('li')[-1]
        # In the case of fimfiction.net, possible statuses are 'Completed', 'Incomplete', 'On Hiatus' and 'Cancelled'
        # For the sake of bringing it in line with the other adapters, 'Incomplete' and 'On Hiatus' become 'In-Progress'
        # and 'Complete' beomes 'Completed'. 'Cancelled' seems an important enough (not to mention more strictly true) 
        # status to leave unchanged.
-        status = status_bar.text.split("|")[0].strip().replace("Incomplete", "In-Progress").replace("On Hiatus", "In-Progress").replace("Complete", "Completed")
-        self.story.setMetadata('status', status)
-        self.story.setMetadata('rating', status_bar.span.text)
-        # This way is less elegant, perhaps, but more robust in face of format changes.
-        numWords = status_bar.find("div",{"class":"word_count"}).b.text
-        self.story.setMetadata('numWords', numWords)
-        
-        description_soup = soup.find("div", {"class":"description"})
-        # Sometimes the description has an expanding element
-        # This removes the ellipsis and the expand button
-        try:
-            description_soup.find('span', {"id":re.compile(r"description_more_elipses_\d+")}).extract() # Web designer can't spell 'ellipsis'
-            description_soup.find('a', {"class":"more"}).extract()
-        except:
-            pass
-        self.story.setMetadata('description', description_soup.text)
-        
-        # Unfortunately, nowhere on the page is the year mentioned.
-        # Best effort to deal with this:
-        # Use this year, if that's a date in the future, subtract one year.
-        # Their earliest story is Jun, so they'll probably change the date
-        # around then.
+        status = storyMetadata["status"].replace("Incomplete", "In-Progress").replace("On Hiatus", "In-Progress").replace("Complete", "Completed")
+        self.story.setMetadata("status", status)
+        self.story.setMetadata("rating", storyMetadata["content_rating_text"])
+            
+        for category in storyMetadata["categories"]:
+            if storyMetadata["categories"][category]:
+                self.story.addToList("genre", category) 

-        now = datetime.datetime.now()
+        self.story.addToList("category", "My Little Pony")
        
-        # Get the date of creation from the first chapter
-        datePublished_text = chapterDates[0]
-        day, month = datePublished_text.split()
-        day = re.sub(r"[^\d.]+", '', day)
-        datePublished = makeDate("%s%s%s"%(now.year,month,day), "%Y%b%d")
-        if datePublished > now :
-            datePublished = datePublished.replace(year=now.year-1)
-        self.story.setMetadata("datePublished", datePublished)
-        dateUpdated_soup = bs.BeautifulSoup(data).find("div", {"class":"calendar"})
-        dateUpdated_soup.find('span').extract()
-        dateUpdated = makeDate("%s%s"%(now.year,dateUpdated_soup.text), "%Y%b%d")
-        if dateUpdated > now :
-            dateUpdated = datePublished.replace(year=now.year-1)
-        self.story.setMetadata("dateUpdated", dateUpdated)
+        self.story.setMetadata("numWords", str(storyMetadata["words"]))
        
+        
+        # fimfic is the first site with an explicit cover image.
+        if self.getConfig('include_images') and "image" in storyMetadata.keys():
+            coverurl = storyMetadata["image"]
+            if coverurl.startswith('//static.fimfiction.net'): # fix for img urls missing 'http:'
+                coverurl = "http:"+coverurl
+            self.story.addImgUrl(self,self.url,coverurl,self._fetchUrlRaw,cover=True)
+            
+        self.setDescription(self.url, storyMetadata["description"])
+        
+        # Dates are in Unix time
+        # Take the publish date from the first chapter posted
+        rawDatePublished = storyMetadata["chapters"][0]["date_modified"]
+        self.story.setMetadata("datePublished", datetime.fromtimestamp(rawDatePublished))
+        rawDateUpdated = storyMetadata["date_modified"]
+        self.story.setMetadata("dateUpdated", datetime.fromtimestamp(rawDateUpdated))
+        
+        soup = bs.BeautifulSoup(data).find("div", {"class":"story"})
+        for character in [character_icon["title"] for character_icon in soup.findAll("a", {"class":"character_icon"})]:
+            self.story.addToList("characters", character)
+            
+            
    def getChapterText(self, url):
        logging.debug('Getting chapter text from: %s' % url)
        soup = bs.BeautifulSoup(self._fetchUrl(url),selfClosingTags=('br','hr')).find('div', {'id' : 'chapter_container'})
        if soup == None:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
-        return utf8FromSoup(soup)
+        return self.utf8FromSoup(url,soup)
        
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class HarryPotterFanFictionComSiteAdapter(BaseSiteAdapter):

@@ -125,7 +125,8 @@ class HarryPotterFanFictionComSiteAdapter(BaseSiteAdapter):

        ## Finding the metadata is a bit of a pain.  Desc is the only thing this color.
        desctable= soup.find('table',{'bgcolor':'#f0e8e8'})
-        self.story.setMetadata('description',stripHTML(desctable))
+        self.setDescription(url,desctable)
+        #self.story.setMetadata('description',stripHTML(desctable))

        ## Finding the metadata is a bit of a pain.  Most of the meta
        ## data is in a center.table without a bgcolor.
@@ -193,7 +194,7 @@ class HarryPotterFanFictionComSiteAdapter(BaseSiteAdapter):
        if None == div:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)

-        return utf8FromSoup(div)
+        return self.utf8FromSoup(url,div)

 def getClass():
    return HarryPotterFanFictionComSiteAdapter
@@ -0,0 +1,234 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+# This function is called by the downloader in all adapter_*.py files
+# in this dir to register the adapter class.  So it needs to be
+# updated to reflect the class below it.  That, plus getSiteDomain()
+# take care of 'Registering'.
+def getClass():
+    return HPFandomNetAdapterAdapter # XXX
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class HPFandomNetAdapterAdapter(BaseSiteAdapter): # XXX
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
+        self.password = ""
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        # XXX Most sites don't have the /eff part.  Replace all to remove it usually.
+        self._setURL('http://' + self.getSiteDomain() + '/eff/viewstory.php?sid='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','hpfdm') # XXX
+
+        # If all stories from the site fall into the same category,
+        # the site itself isn't likely to label them as such, so we
+        # do.
+        self.story.addToList("category","Harry Potter") # XXX
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%Y.%m.%d" # XXX
+            
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'www.hpfandom.net' # XXX
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/eff/viewstory.php?sid=1234"
+
+    def getSiteURLPattern(self):
+        return re.escape("http://"+self.getSiteDomain()+"/eff/viewstory.php?sid=")+r"\d+$"
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        url = self.url
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        # print data
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        # Find authorid and URL from... author url.
+        a = soup.find('a', href=re.compile(r"viewuser.php\?uid=\d+"))
+        self.story.setMetadata('authorId',a['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/eff/'+a['href'])
+        self.story.setMetadata('author',a.string)
+
+        ## Going to get the rest from the author page.
+        authdata = self._fetchUrl(self.story.getMetadata('authorUrl'))
+        # fix a typo in the site HTML so I can find the Characters list.
+        authdata = authdata.replace('<td width=10%">','<td width="10%">')
+
+        # hpfandom.net only seems to indicate adult-only by javascript on the story/chapter links.
+        if "javascript:if (confirm('Slash/het fiction which incorporates sexual situations to a somewhat graphic degree and some violence. ')) location = 'viewstory.php?sid=%s'"%self.story.getMetadata('storyId') in authdata \
+                and not (self.is_adult or self.getConfig("is_adult")):
+            raise exceptions.AdultCheckRequired(self.url)
+        
+        authsoup = bs.BeautifulSoup(authdata)
+
+        reviewsa = authsoup.find('a', href="reviews.php?sid="+self.story.getMetadata('storyId')+"&a=")
+        # <table><tr><td><p><b><a ...>
+        metablock = reviewsa.findParent("table")
+        #print("metablock:%s"%metablock)
+        
+        ## Title
+        titlea = metablock.find('a', href=re.compile("viewstory.php"))
+        #print("titlea:%s"%titlea)
+        if titlea == None:
+            raise exceptions.FailedToDownload("Story URL (%s) not found on author's page, can't use chapter URLs"%url)
+        self.story.setMetadata('title',stripHTML(titlea))
+        
+        # Find the chapters: !!! hpfandom.net differs from every other
+        # eFiction site--the sid on viewstory for chapters is
+        # *different* for each chapter
+        for chapter in soup.findAll('a', {'href':re.compile(r"viewstory.php\?sid=\d+&i=\d+")}):
+            m = re.match(r'.*?(viewstory.php\?sid=\d+&i=\d+).*?',chapter['href'])
+            # just in case there's tags, like <i> in chapter titles.
+            #print("====chapter===%s"%m.group(1))
+            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/eff/'+m.group(1)))
+
+        if len(self.chapterUrls) == 0:
+            self.chapterUrls.append((stripHTML(self.story.getMetadata('title')),url))
+
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+
+        summary = metablock.find("td",{"class":"summary"})
+        summary.name='span'
+        self.setDescription(url,summary)
+
+        # words & completed in first row of metablock.
+        firstrow = stripHTML(metablock.find('tr'))
+        # A Mother's Love xx Going Grey 1 (G+) by Kiristeen | Reviews - 18 | Words: 27468 | Completed: Yes
+        m = re.match(r".*?\((?P<rating>[^)]+)\).*?Words: (?P<words>\d+).*?Completed: (?P<status>Yes|No)",firstrow)
+        if m != None:
+            if m.group('rating') != None:
+                self.story.setMetadata('rating', m.group('rating'))
+                
+            if m.group('words') != None:
+                self.story.setMetadata('numWords', m.group('words'))
+
+            if m.group('status') != None:
+                if 'Yes' in m.group('status'):
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+                
+
+        # <tr><td width="10%" valign="top">Chapters:</td><td width="40%" valign="top">4</td>
+        # <td width="10%" valign="top">Published:</td><td width="40%" valign="top">2010.09.29</td></tr>
+        # <tr><td width="10%" valign="top">Completed:</td><td width="40%" valign="top">Yes</td><td width="10%" valign="top">Updated:</td><td width="40%" valign="top">2010.10.03</td></tr>
+        labels = metablock.findAll('td',{'width':'10%'})
+        for td in labels:
+            label = td.string
+            value = td.nextSibling.string
+            #print("\nlabel:%s\nvalue:%s\n"%(label,value))
+
+            if 'Category' in label:
+                cats = td.parent.findAll('a',href=re.compile(r'categories.php'))
+                catstext = [cat.string for cat in cats]
+                for cat in catstext:
+                    self.story.addToList('category',cat.string)
+
+            if 'Characters' in label:
+                for char in value.split(','):
+                    self.story.addToList('characters',char.strip())
+
+            if 'Genre' in label:
+                for genre in value.split(','):
+                    self.story.addToList('genre',genre.strip())
+
+            if 'Warnings' in label:
+                for warning in value.split(','):
+                    if warning.strip() != 'none':
+                        self.story.addToList('warnings',warning.strip())
+
+            if 'Published' in label:
+                self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
+            
+            if 'Updated' in label:
+                self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        data = self._fetchUrl(url)
+        # There's no good wrapper around the chapter text. :-/
+        # There are, however, tables with width=100% just above and below the real text.
+        data = re.sub(r'<table width="100%">.*?</table>','<div name="storybody">',
+                      data,count=1,flags=re.DOTALL)
+        
+        data = re.sub(r'<table width="100%">.*?</table>','</div>',
+                      data,count=1,flags=re.DOTALL)
+        
+        soup = bs.BeautifulStoneSoup(data,selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+
+        div = soup.find("div",{'name':'storybody'})
+        #print("\n\ndiv:%s\n\n"%div)
+        
+        if None == div:
+            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+        return self.utf8FromSoup(url,div)
@@ -0,0 +1,306 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+# Search for XXX comments--that's where things are most likely to need changing.
+
+# This function is called by the downloader in all adapter_*.py files
+# in this dir to register the adapter class.  So it needs to be
+# updated to reflect the class below it.  That, plus getSiteDomain()
+# take care of 'Registering'.
+def getClass():
+    return KSArchiveComAdapter # XXX
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class KSArchiveComAdapter(BaseSiteAdapter): # XXX
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        # XXX Most sites don't have the /fanfic part.  Replace all to remove it usually.
+        self._setURL('http://' + self.getSiteDomain() + '/viewstory.php?sid='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','ksa') # XXX
+
+        # If all stories from the site fall into the same category,
+        # the site itself isn't likely to label them as such, so we
+        # do.
+        self.story.addToList("category","Star Trek") # XXX
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%b/%d/%Y" # XXX
+            
+    @classmethod
+    def getAcceptDomains(cls):
+        return ['www.ksarchive.com','ksarchive.com']
+
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'ksarchive.com' # XXX
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/viewstory.php?sid=1234"
+
+    def getSiteURLPattern(self):
+        return "http://(www.)?"+re.escape(self.getSiteDomain()+"/viewstory.php?sid=")+r"\d+$"
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        if self.is_adult or self.getConfig("is_adult"):
+            # Weirdly, different sites use different warning numbers.
+            # If the title search below fails, there's a good chance
+            # you need a different number.  print data at that point
+            # and see what the 'click here to continue' url says.
+
+            # Furthermore, there's a couple sites now with more than
+            # one warning level for different ratings.  And they're
+            # fussy about it.  midnightwhispers has three: 10, 3 & 5.
+            # we'll try 5 first.
+            addurl = "&ageconsent=ok&warning=2" # XXX
+        else:
+            addurl=""
+
+        # index=1 makes sure we see the story chapter index.  Some
+        # sites skip that for one-chapter stories.
+        url = self.url+'&index=1'+addurl
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        # The actual text that is used to announce you need to be an
+        # adult varies from site to site.  Again, print data before
+        # the title search to troubleshoot.
+            
+        # Since the warning text can change by warning level, let's
+        # look for the warning pass url.  ksarchive uses
+        # &amp;warning= -- actually, so do other sites.  Must be an
+        # eFiction book.
+            
+        # viewstory.php?sid=1882&amp;warning=4
+        # viewstory.php?sid=1654&amp;ageconsent=ok&amp;warning=5
+        #print data
+        #m = re.search(r"'viewstory.php\?sid=1882(&amp;warning=4)'",data)
+        m = re.search(r"'viewstory.php\?sid=\d+((?:&amp;ageconsent=ok)?&amp;warning=\d+)'",data)
+        if m != None:
+            if self.is_adult or self.getConfig("is_adult"):
+                # We tried the default and still got a warning, so
+                # let's pull the warning number from the 'continue'
+                # link and reload data.
+                addurl = m.group(1)
+                # correct stupid &amp; error in url.
+                addurl = addurl.replace("&amp;","&")
+                url = self.url+'&index=1'+addurl
+                logging.debug("URL 2nd try: "+url)
+
+                try:
+                    data = self._fetchUrl(url)
+                except urllib2.HTTPError, e:
+                    if e.code == 404:
+                        raise exceptions.StoryDoesNotExist(self.url)
+                    else:
+                        raise e    
+            else:
+                raise exceptions.AdultCheckRequired(self.url)
+            
+        if "Access denied. This story has not been validated by the adminstrators of this site." in data:
+            raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        # print data
+
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        ## Title
+        a = soup.find('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"$"))
+        self.story.setMetadata('title',stripHTML(a)) # title's inside a <b> tag.
+
+        # Find authorid and URL from... author url.
+        a = soup.find('a', href=re.compile(r"viewuser.php\?uid=\d+"))
+        self.story.setMetadata('authorId',a['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/'+a['href'])
+        self.story.setMetadata('author',a.string)
+
+        # Find the chapters:
+        for chapter in soup.findAll('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"&chapter=\d+$")):
+            # just in case there's tags, like <i> in chapter titles.
+            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/'+chapter['href']+addurl))
+
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+
+        # <span class="label">Rated:</span> NC-17<br /> etc
+        labels = soup.findAll('span',{'class':'label'})
+        for labelspan in labels:
+            value = labelspan.nextSibling
+            label = labelspan.string
+
+            if 'Summary' in label:
+                ## Everything until the next span class='label'
+                svalue = ""
+                while not defaultGetattr(value,'class') == 'label':
+                    svalue += str(value)
+                    value = value.nextSibling
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))
+
+            if 'Rated' in label:
+                self.story.setMetadata('rating', value)
+
+            if 'Word count' in label:
+                self.story.setMetadata('numWords', value)
+
+            if 'Categories' in label:
+                cats = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=categories'))
+                catstext = [cat.string for cat in cats]
+                for cat in catstext:
+                    if cat.string.strip() in ('Poetry','Essays'):
+                        self.story.addToList('category',cat.string)
+
+            if 'Characters' in label:
+                self.story.addToList('characters','Kirk')
+                self.story.addToList('characters','Spock')
+                chars = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=characters'))
+                charstext = [char.string for char in chars]
+                for char in charstext:
+                    self.story.addToList('characters',char.string)
+
+            ## Not all sites use Genre, but there's no harm to
+            ## leaving it in.  Check to make sure the type_id number
+            ## is correct, though--it's site specific.
+            if 'Genre' in label:
+                genres = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=1')) # XXX
+                genrestext = [genre.string for genre in genres]
+                self.genre = ', '.join(genrestext)
+                for genre in genrestext:
+                    self.story.addToList('genre',genre.string)
+
+            ## In addition to Genre (which is very site specific) KSA
+            ## has 'Story Type', which is much more what most sites
+            ## call genre.
+            if 'Story Type' in label:
+                genres = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=5')) # XXX
+                genrestext = [genre.string for genre in genres]
+                self.genre = ', '.join(genrestext)
+                for genre in genrestext:
+                    self.story.addToList('genre',genre.string)
+
+            ## Not all sites use Warnings, but there's no harm to
+            ## leaving it in.  Check to make sure the type_id number
+            ## is correct, though--it's site specific.
+            if 'Warnings' in label:
+                warnings = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2')) # XXX
+                warningstext = [warning.string for warning in warnings]
+                self.warning = ', '.join(warningstext)
+                for warning in warningstext:
+                    self.story.addToList('warnings',warning.string)
+
+            if 'Completed' in label:
+                if 'Yes' in value:
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+
+            if 'Published' in label:
+                self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
+            
+            if 'Updated' in label:
+                # there's a stray [ at the end.
+                #value = value[0:-1]
+                self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
+
+        try:
+            # Find Series name from series URL.
+            a = soup.find('a', href=re.compile(r"viewseries.php\?seriesid=\d+"))
+            series_name = a.string
+            series_url = 'http://'+self.host+'/'+a['href']
+
+            # use BeautifulSoup HTML parser to make everything easier to find.
+            seriessoup = bs.BeautifulSoup(self._fetchUrl(series_url))
+            storyas = seriessoup.findAll('a', href=re.compile(r'^viewstory.php\?sid=\d+$'))
+            i=1
+            for a in storyas:
+                if a['href'] == ('viewstory.php?sid='+self.story.getMetadata('storyId')):
+                    self.setSeries(series_name, i)
+                    break
+                i+=1
+            
+        except:
+            # I find it hard to care if the series parsing fails
+            pass
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        data = self._fetchUrl(url)
+        soup = bs.BeautifulStoneSoup(data,
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+        
+        div = soup.find('div', {'id' : 'story'})
+
+        if None == div:
+            if "A fatal MySQL error was encountered" in data:
+                raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Database error on the site reported!" % url)
+            else:
+                raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,div)
@@ -0,0 +1,250 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+
+def getClass():
+    return LibraryOfMoriaComAdapter
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class LibraryOfMoriaComAdapter(BaseSiteAdapter):
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
+        self.password = ""
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        self._setURL('http://' + self.getSiteDomain() + '/a/viewstory.php?sid='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','lom')
+
+        # If all stories from the site fall into the same category,
+        # the site itself isn't likely to label them as such, so we
+        # do.
+        self.story.addToList("category","Lord of the Rings")
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%B %d, %Y"
+            
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'www.libraryofmoria.com'
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/a/viewstory.php?sid=1234"
+
+    def getSiteURLPattern(self):
+        return re.escape("http://"+self.getSiteDomain()+"/a/viewstory.php?sid=")+r"\d+$"
+
+
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        if self.is_adult or self.getConfig("is_adult"):
+            addurl = "&ageconsent=ok&warning=3"
+        else:
+            addurl=""
+
+        # index=1 makes sure we see the story chapter index.  Some
+        # sites skip that for one-chapter stories.
+        url = self.url+addurl
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        m = re.search(r"'viewstory.php\?sid=\d+((?:&amp;ageconsent=ok)?&amp;warning=\d+)'",data)
+        if m != None:
+            if self.is_adult or self.getConfig("is_adult"):
+                # We tried the default and still got a warning, so
+                # let's pull the warning number from the 'continue'
+                # link and reload data.
+                addurl = m.group(1)
+                # correct stupid &amp; error in url.
+                addurl = addurl.replace("&amp;","&")
+                url = self.url+'&index=1'+addurl
+                logging.debug("URL 2nd try: "+url)
+
+                try:
+                    data = self._fetchUrl(url)
+                except urllib2.HTTPError, e:
+                    if e.code == 404:
+                        raise exceptions.StoryDoesNotExist(self.url)
+                    else:
+                        raise e    
+            else:
+                raise exceptions.AdultCheckRequired(self.url)
+            
+        if "Access denied. This story has not been validated by the adminstrators of this site." in data:
+            raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        # print data
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        ## Title
+        a = soup.find('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"$"))
+        self.story.setMetadata('title',a.string)
+        
+        # Find authorid and URL from... author url.
+        a = soup.find('a', href=re.compile(r"viewuser.php\?uid=\d+"))
+        self.story.setMetadata('authorId',a['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/'+a['href'])
+        self.story.setMetadata('author',a.string)
+
+        # Find the chapters:
+        for chapter in soup.findAll('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"&chapter=\d+$")):
+            # just in case there's tags, like <i> in chapter titles.
+            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/a/'+chapter['href']+addurl))
+
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+
+        # <span class="label">Rated:</span> NC-17<br /> etc
+        labels = soup.findAll('span',{'class':'label'})
+        for labelspan in labels:
+            value = labelspan.nextSibling
+            label = labelspan.string
+
+            if 'Summary' in label:
+                ## Everything until the next span class='label'
+                svalue = ""
+                while not defaultGetattr(value,'class') == 'label':
+                    svalue += str(value)
+                    value = value.nextSibling
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))
+
+            if 'Rated' in label:
+                self.story.setMetadata('rating', value)
+
+            if 'Word count' in label:
+                self.story.setMetadata('numWords', value)
+
+            if 'Categories' in label:
+                cats = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=categories'))
+                catstext = [cat.string for cat in cats]
+                for cat in catstext:
+                    self.story.addToList('category',cat.string)
+
+            if 'Characters' in label:
+                chars = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=characters'))
+                charstext = [char.string for char in chars]
+                for char in charstext:
+                    self.story.addToList('characters',char.string)
+
+            if 'Type' in label:
+                genres = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=1'))
+                for genre in genres:
+                    self.story.addToList('genre',genre.string)
+
+            if 'Warning' in label:
+                warnings = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=5'))
+                for warning in warnings:
+                    self.story.addToList('warnings',warning.string)
+
+            if 'Completed' in label:
+                if 'Yes' in value:
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+
+            if 'Published' in label:
+                self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
+            
+            if 'Updated' in label:
+                # there's a stray [ at the end.
+                #value = value[0:-1]
+                self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
+
+        try:
+            # Find Series name from series URL.
+            a = soup.find('a', href=re.compile(r"viewseries.php\?seriesid=\d+"))
+            series_name = a.string
+            series_url = 'http://'+self.host+'/a/'+a['href']
+
+            # use BeautifulSoup HTML parser to make everything easier to find.
+            seriessoup = bs.BeautifulSoup(self._fetchUrl(series_url))
+            storyas = seriessoup.findAll('a', href=re.compile(r'^viewstory.php\?sid=\d+$'))
+            i=1
+            for a in storyas:
+                if a['href'] == ('viewstory.php?sid='+self.story.getMetadata('storyId')):
+                    self.setSeries(series_name, i)
+                    break
+                i+=1
+            
+        except:
+            # I find it hard to care if the series parsing fails
+            pass
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        soup = bs.BeautifulStoneSoup(self._fetchUrl(url),
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+        
+        div = soup.find('div', {'id' : 'story'})
+
+        if None == div:
+            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,div)
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class MediaMinerOrgSiteAdapter(BaseSiteAdapter):

@@ -174,7 +174,8 @@ class MediaMinerOrgSiteAdapter(BaseSiteAdapter):
        # Summary: ....
        m = re.match(r".*?Summary: (.*)$",metastr) 
        if m:
-            self.story.setMetadata('description', m.group(1))
+            self.setDescription(url, m.group(1))
+            #self.story.setMetadata('description', m.group(1))

        # completed
        m = re.match(r".*?Status: Completed.*?",metastr)
@@ -210,7 +211,7 @@ class MediaMinerOrgSiteAdapter(BaseSiteAdapter):
                del div['style']
                del div['align']
            anchor.name='div'
-            return utf8FromSoup(anchor)
+            return self.utf8FromSoup(url,anchor)
        
        else:
            logging.debug('Using kludgey text find for older mediaminer story.')
@@ -226,7 +227,7 @@ class MediaMinerOrgSiteAdapter(BaseSiteAdapter):
                    soup.findAll('table',{'class':'tbbrdr'}):
                tag.extract() # remove tag from soup.
                
-            return utf8FromSoup(soup)
+            return self.utf8FromSoup(url,soup)
        

 def getClass():
@@ -0,0 +1,289 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+# Search for XXX comments--that's where things are most likely to need changing.
+
+# This function is called by the downloader in all adapter_*.py files
+# in this dir to register the adapter class.  So it needs to be
+# updated to reflect the class below it.  That, plus getSiteDomain()
+# take care of 'Registering'.
+def getClass():
+    return MidnightwhispersCaAdapter # XXX
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class MidnightwhispersCaAdapter(BaseSiteAdapter): # XXX
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        # XXX Most sites don't have the /fanfic part.  Replace all to remove it usually.
+        self._setURL('http://' + self.getSiteDomain() + '/viewstory.php?sid='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','mw') # XXX
+
+        # If all stories from the site fall into the same category,
+        # the site itself isn't likely to label them as such, so we
+        # do.
+        self.story.addToList("category","Queer as Folk") # XXX
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%B %d, %Y" # XXX
+            
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'www.midnightwhispers.ca' # XXX
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/viewstory.php?sid=1234"
+
+    def getSiteURLPattern(self):
+        return re.escape("http://"+self.getSiteDomain()+"/viewstory.php?sid=")+r"\d+$"
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        if self.is_adult or self.getConfig("is_adult"):
+            # Weirdly, different sites use different warning numbers.
+            # If the title search below fails, there's a good chance
+            # you need a different number.  print data at that point
+            # and see what the 'click here to continue' url says.
+
+            # Furthermore, there's a couple sites now with more than
+            # one warning level for different ratings.  And they're
+            # fussy about it.  midnightwhispers has three: 10, 3 & 5.
+            # we'll try 5 first.
+            addurl = "&ageconsent=ok&warning=5" # XXX
+        else:
+            addurl=""
+
+        # index=1 makes sure we see the story chapter index.  Some
+        # sites skip that for one-chapter stories.
+        url = self.url+'&index=1'+addurl
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        # The actual text that is used to announce you need to be an
+        # adult varies from site to site.  Again, print data before
+        # the title search to troubleshoot.
+            
+        # Since the warning text can change by warning level, let's
+        # look for the warning pass url.  nfacommunity uses
+        # &amp;warning= -- actually, so do other sites.  Must be an
+        # eFiction book.
+            
+        # viewstory.php?sid=1882&amp;warning=4
+        # viewstory.php?sid=1654&amp;ageconsent=ok&amp;warning=5
+        #print data
+        #m = re.search(r"'viewstory.php\?sid=1882(&amp;warning=4)'",data)
+        m = re.search(r"'viewstory.php\?sid=\d+((?:&amp;ageconsent=ok)?&amp;warning=\d+)'",data)
+        if m != None:
+            if self.is_adult or self.getConfig("is_adult"):
+                # We tried the default and still got a warning, so
+                # let's pull the warning number from the 'continue'
+                # link and reload data.
+                addurl = m.group(1)
+                # correct stupid &amp; error in url.
+                addurl = addurl.replace("&amp;","&")
+                url = self.url+'&index=1'+addurl
+                logging.debug("URL 2nd try: "+url)
+
+                try:
+                    data = self._fetchUrl(url)
+                except urllib2.HTTPError, e:
+                    if e.code == 404:
+                        raise exceptions.StoryDoesNotExist(self.url)
+                    else:
+                        raise e    
+            else:
+                raise exceptions.AdultCheckRequired(self.url)
+            
+        if "Access denied. This story has not been validated by the adminstrators of this site." in data:
+            raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        # print data
+
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        ## Title
+        a = soup.find('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"$"))
+        self.story.setMetadata('title',stripHTML(a)) # title's inside a <b> tag.
+
+        # Find authorid and URL from... author url.
+        a = soup.find('a', href=re.compile(r"viewuser.php\?uid=\d+"))
+        self.story.setMetadata('authorId',a['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/'+a['href'])
+        self.story.setMetadata('author',a.string)
+
+        # Find the chapters:
+        for chapter in soup.findAll('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"&chapter=\d+$")):
+            # just in case there's tags, like <i> in chapter titles.
+            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/'+chapter['href']+addurl))
+
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+
+        # <span class="label">Rated:</span> NC-17<br /> etc
+        labels = soup.findAll('span',{'class':'label'})
+        for labelspan in labels:
+            value = labelspan.nextSibling
+            label = labelspan.string
+
+            if 'Summary' in label:
+                ## Everything until the next span class='label'
+                svalue = ""
+                while not defaultGetattr(value,'class') == 'label':
+                    svalue += str(value)
+                    value = value.nextSibling
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))
+
+            if 'Rated' in label:
+                self.story.setMetadata('rating', value)
+
+            if 'Word count' in label:
+                self.story.setMetadata('numWords', value)
+
+            if 'Categories' in label:
+                cats = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=categories'))
+                catstext = [cat.string for cat in cats]
+                for cat in catstext:
+                    self.story.addToList('category',cat.string)
+
+            if 'Characters' in label:
+                chars = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=characters'))
+                charstext = [char.string for char in chars]
+                for char in charstext:
+                    self.story.addToList('characters',char.string)
+
+            ## Not all sites use Genre, but there's no harm to
+            ## leaving it in.  Check to make sure the type_id number
+            ## is correct, though--it's site specific.
+            if 'Genre' in label:
+                genres = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2')) # XXX
+                genrestext = [genre.string for genre in genres]
+                self.genre = ', '.join(genrestext)
+                for genre in genrestext:
+                    self.story.addToList('genre',genre.string)
+
+            ## Not all sites use Warnings, but there's no harm to
+            ## leaving it in.  Check to make sure the type_id number
+            ## is correct, though--it's site specific.
+            if 'Warnings' in label:
+                warnings = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2')) # XXX
+                warningstext = [warning.string for warning in warnings]
+                self.warning = ', '.join(warningstext)
+                for warning in warningstext:
+                    self.story.addToList('warnings',warning.string)
+
+            if 'Completed' in label:
+                if 'Yes' in value:
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+
+            if 'Published' in label:
+                self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
+            
+            if 'Updated' in label:
+                # there's a stray [ at the end.
+                #value = value[0:-1]
+                self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
+
+        try:
+            # Find Series name from series URL.
+            a = soup.find('a', href=re.compile(r"viewseries.php\?seriesid=\d+"))
+            series_name = a.string
+            series_url = 'http://'+self.host+'/'+a['href']
+
+            # use BeautifulSoup HTML parser to make everything easier to find.
+            seriessoup = bs.BeautifulSoup(self._fetchUrl(series_url))
+            storyas = seriessoup.findAll('a', href=re.compile(r'^viewstory.php\?sid=\d+$'))
+            i=1
+            for a in storyas:
+                if a['href'] == ('viewstory.php?sid='+self.story.getMetadata('storyId')):
+                    self.setSeries(series_name, i)
+                    break
+                i+=1
+            
+        except:
+            # I find it hard to care if the series parsing fails
+            pass
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        data = self._fetchUrl(url)
+        soup = bs.BeautifulStoneSoup(data,
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+        
+        div = soup.find('div', {'id' : 'story'})
+
+        if None == div:
+            if "A fatal MySQL error was encountered" in data:
+                raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Database error on the site reported!" % url)
+            else:
+                raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,div)
@@ -0,0 +1,331 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+# This function is called by the downloader in all adapter_*.py files
+# in this dir to register the adapter class.  So it needs to be
+# updated to reflect the class below it.  That, plus getSiteDomain()
+# take care of 'Registering'.
+def getClass():
+    return MuggleNetComAdapter # XXX
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class MuggleNetComAdapter(BaseSiteAdapter): # XXX
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
+        self.password = ""
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        self._setURL('http://' + self.getSiteDomain() + '/viewstory.php?sid='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','mgln') # XXX
+
+        # If all stories from the site fall into the same category,
+        # the site itself isn't likely to label them as such, so we
+        # do.
+        self.story.addToList("category","Harry Potter") # XXX
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%m/%d/%y" # XXX
+            
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'fanfiction.mugglenet.com' # XXX
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/viewstory.php?sid=1234"
+
+    def getSiteURLPattern(self):
+        return re.escape("http://"+self.getSiteDomain()+"/viewstory.php?sid=")+r"\d+$"
+
+    ## Login seems to be reasonably standard across eFiction sites.
+    def needToLoginCheck(self, data):
+        if "class='errortext'>Registered Users Only" in data \
+                or 'There is no such account on our website' in data \
+                or "That password doesn't match the one in our database" in data:
+            return True
+        else:
+            return False
+        
+    def performLogin(self, url):
+        params = {}
+
+        if self.password:
+            params['penname'] = self.username
+            params['password'] = self.password
+        else:
+            params['penname'] = self.getConfig("username")
+            params['password'] = self.getConfig("password")
+        params['cookiecheck'] = '1'
+        params['submit'] = 'Submit'
+    
+        loginUrl = 'http://' + self.getSiteDomain() + '/user.php?action=login&sid='+self.story.getMetadata('storyId')
+        logging.debug("Will now login to URL (%s) as (%s)" % (loginUrl,
+                                                              params['penname']))
+    
+        d = self._fetchUrl(loginUrl, params)
+    
+        if "Member Account" not in d : #Member Account
+            logging.info("Failed to login to URL %s as %s" % (loginUrl,
+                                                              params['penname']))
+            raise exceptions.FailedToLogin(url,params['penname'])
+            return False
+        else:
+            return True
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        if self.is_adult or self.getConfig("is_adult"):
+            # Weirdly, different sites use different warning numbers.
+            # If the title search below fails, there's a good chance
+            # you need a different number.  print data at that point
+            # and see what the 'click here to continue' url says.
+            # http://fanfiction.mugglenet.com/viewstory.php?sid=91079&ageconsent=ok&warning=3
+            addurl = "&ageconsent=ok&warning=3" # XXX  &warning=5
+        else:
+            addurl=""
+
+        # index=1 makes sure we see the story chapter index.  Some
+        # sites skip that for one-chapter stories.
+        url = self.url+'&index=1'+addurl
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        if self.needToLoginCheck(data):
+            # need to log in for this one.
+            self.performLogin(url)
+            data = self._fetchUrl(url)
+            #print("\nurl:%s\ndata:\n%s\n"%(url,data))
+
+        # The actual text that is used to announce you need to be an
+        # adult varies from site to site.  Again, print data before
+        # the title search to troubleshoot.
+            
+        # Since the warning text can change by warning level, let's
+        # look for the warning pass url.  nfacommunity uses
+        # &amp;warning= -- actually, so do other sites.  Must be an
+        # eFiction book.
+            
+        # viewstory.php?sid=1882&amp;warning=4
+        # viewstory.php?sid=1654&amp;ageconsent=ok&amp;warning=5
+        #print data
+        #m = re.search(r"'viewstory.php\?sid=1882(&amp;warning=4)'",data)
+        m = re.search(r"'viewstory.php\?sid=%s((?:&amp;ageconsent=ok)?&amp;warning=\d+)'"%self.story.getMetadata('storyId'),data)
+        if m != None:
+            if self.is_adult or self.getConfig("is_adult"):
+                # We tried the default and still got a warning, so
+                # let's pull the warning number from the 'continue'
+                # link and reload data.
+                addurl = m.group(1)
+                # correct stupid &amp; error in url.
+                addurl = addurl.replace("&amp;","&")
+                url = self.url+'&index=1'+addurl
+                logging.debug("URL 2nd try: "+url)
+
+                try:
+                    data = self._fetchUrl(url)
+                except urllib2.HTTPError, e:
+                    if e.code == 404:
+                        raise exceptions.StoryDoesNotExist(self.url)
+                    else:
+                        raise e    
+            else:
+                raise exceptions.AdultCheckRequired(self.url)
+            
+        if "Access denied. This story has not been validated by the adminstrators of this site." in data:
+            raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        # print data
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        ## Title
+        a = soup.find('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"$"))
+        self.story.setMetadata('title',a.string)
+        
+        # Find authorid and URL from... author url.
+        a = soup.find('a', href=re.compile(r"viewuser.php\?uid=\d+"))
+        self.story.setMetadata('authorId',a['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/'+a['href'])
+        self.story.setMetadata('author',a.string)
+
+        # Find the chapters:
+        for chapter in soup.findAll('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"&chapter=\d+$")):
+            # just in case there's tags, like <i> in chapter titles.
+            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/'+chapter['href']+addurl))
+
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+
+        # Not good enough-- content can contain a ('), which ends the content prematurely.
+        # metadesc = soup.find('meta',{'name':'description'})
+        # print("removeAllEntities(metadesc['content']):\n%s\n"%removeAllEntities(metadesc['content']))
+        start='<span class="label">Summary: </span>'
+        end='<span class="label">Rated:</span>'
+        summarydata = data[data.index(start)+len(start):data.index(end)]
+        #print("summarydata:\n%s\n"%summarydata)
+        self.setDescription(url,bs.BeautifulSoup(summarydata))
+        
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+
+        # <span class="label">Rated:</span> NC-17<br /> etc
+        labels = soup.findAll('span',{'class':'label'})
+        for labelspan in labels:
+            value = labelspan.nextSibling
+            label = labelspan.string
+
+            # not good enough--poorly formated summary html will break it.
+            # if 'Summary' in label:
+            #     ## Everything until the next span class='label'
+            #     svalue = ""
+            #     while not defaultGetattr(value,'class') == 'label':
+            #         svalue += str(value)
+            #         value = value.nextSibling
+            #     self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))
+
+            if 'Rated' in label:
+                self.story.setMetadata('rating', value)
+
+            if 'Word count' in label:
+                self.story.setMetadata('numWords', value)
+
+            if 'Categories' in label:
+                cats = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=categories'))
+                catstext = [cat.string for cat in cats]
+                for cat in catstext:
+                    self.story.addToList('category',cat.string)
+
+            if 'Characters' in label:
+                chars = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=characters'))
+                charstext = [char.string for char in chars]
+                for char in charstext:
+                    self.story.addToList('characters',char.string)
+
+            ## Not all sites use Genre, but there's no harm to
+            ## leaving it in.  Check to make sure the type_id number
+            ## is correct, though--it's site specific.
+            if 'Genre' in label:
+                genres = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2')) # XXX
+                genrestext = [genre.string for genre in genres]
+                self.genre = ', '.join(genrestext)
+                for genre in genrestext:
+                    self.story.addToList('genre',genre.string)
+
+            ## Not all sites use Warnings, but there's no harm to
+            ## leaving it in.  Check to make sure the type_id number
+            ## is correct, though--it's site specific.
+            if 'Warnings' in label:
+                warnings = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2')) # XXX
+                warningstext = [warning.string for warning in warnings]
+                self.warning = ', '.join(warningstext)
+                for warning in warningstext:
+                    self.story.addToList('warnings',warning.string)
+
+            if 'Completed' in label:
+                if 'Yes' in value:
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+
+            if 'Published' in label:
+                self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
+            
+            if 'Updated' in label:
+                # there's a stray [ at the end.
+                #value = value[0:-1]
+                self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
+
+        try:
+            # Find Series name from series URL.
+            a = soup.find('a', href=re.compile(r"viewseries.php\?seriesid=\d+"))
+            series_name = a.string
+            series_url = 'http://'+self.host+'/'+a['href']
+
+            # use BeautifulSoup HTML parser to make everything easier to find.
+            seriessoup = bs.BeautifulSoup(self._fetchUrl(series_url))
+            storyas = seriessoup.findAll('a', href=re.compile(r'^viewstory.php\?sid=\d+$'))
+            i=1
+            for a in storyas:
+                if a['href'] == ('viewstory.php?sid='+self.story.getMetadata('storyId')):
+                    self.setSeries(series_name, i)
+                    break
+                i+=1
+            
+        except:
+            # I find it hard to care if the series parsing fails
+            pass
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        soup = bs.BeautifulStoneSoup(self._fetchUrl(url),
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+        
+        div = soup.find('div', {'id' : 'story'})
+
+        if None == div:
+            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,div)
@@ -0,0 +1,289 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+# Search for XXX comments--that's where things are most likely to need changing.
+
+# This function is called by the downloader in all adapter_*.py files
+# in this dir to register the adapter class.  So it needs to be
+# updated to reflect the class below it.  That, plus getSiteDomain()
+# take care of 'Registering'.
+def getClass():
+    return NfaCommunityComAdapter # XXX
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class NfaCommunityComAdapter(BaseSiteAdapter): # XXX
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        # XXX Most sites don't have the /fanfic part.  Replace all to remove it usually.
+        self._setURL('http://' + self.getSiteDomain() + '/viewstory.php?sid='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','nfa') # XXX
+
+        # If all stories from the site fall into the same category,
+        # the site itself isn't likely to label them as such, so we
+        # do.
+        self.story.addToList("category","NCIS") # XXX
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%m/%d/%Y" # XXX
+            
+    @classmethod
+    def getAcceptDomains(cls):
+        return ['www.nfacommunity.com','nfacommunity.com']
+
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'nfacommunity.com' # XXX
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/viewstory.php?sid=1234"
+
+    def getSiteURLPattern(self):
+        return "http://(www.)?"+re.escape(self.getSiteDomain()+"/viewstory.php?sid=")+r"\d+$"
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        if self.is_adult or self.getConfig("is_adult"):
+            # Weirdly, different sites use different warning numbers.
+            # If the title search below fails, there's a good chance
+            # you need a different number.  print data at that point
+            # and see what the 'click here to continue' url says.
+
+            # Furthermore, there's a couple sites now with more than
+            # one warning level for different ratings.  And they're
+            # fussy about it.  nfacommunity has two: 4 & 5.
+            # we'll try 5 first.
+            addurl = "&ageconsent=ok&warning=5" # XXX
+        else:
+            addurl=""
+
+        # index=1 makes sure we see the story chapter index.  Some
+        # sites skip that for one-chapter stories.
+        url = self.url+'&index=1'+addurl
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        # The actual text that is used to announce you need to be an
+        # adult varies from site to site.  Again, print data before
+        # the title search to troubleshoot.
+            
+        # Since the warning text can change by warning level, let's
+        # look for the warning pass url.  nfacommunity uses
+        # &amp;warning= -- actually, so do other sites.  Must be an
+        # eFiction book.
+            
+        # viewstory.php?sid=1882&amp;warning=4
+        # viewstory.php?sid=1654&amp;ageconsent=ok&amp;warning=5
+        #print data
+        #m = re.search(r"'viewstory.php\?sid=1882(&amp;warning=4)'",data)
+        m = re.search(r"'viewstory.php\?sid=\d+((?:&amp;ageconsent=ok)?&amp;warning=\d+)'",data)
+        if m != None:
+            if self.is_adult or self.getConfig("is_adult"):
+                # We tried the default and still got a warning, so
+                # let's pull the warning number from the 'continue'
+                # link and reload data.
+                addurl = m.group(1)
+                # correct stupid &amp; error in url.
+                addurl = addurl.replace("&amp;","&")
+                url = self.url+'&index=1'+addurl
+                logging.debug("URL 2nd try: "+url)
+
+                try:
+                    data = self._fetchUrl(url)
+                except urllib2.HTTPError, e:
+                    if e.code == 404:
+                        raise exceptions.StoryDoesNotExist(self.url)
+                    else:
+                        raise e    
+            else:
+                raise exceptions.AdultCheckRequired(self.url)
+            
+        if "Access denied. This story has not been validated by the adminstrators of this site." in data:
+            raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        ## Title
+        a = soup.find('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"$"))
+        self.story.setMetadata('title',a.string)
+        
+        # Find authorid and URL from... author url.
+        a = soup.find('a', href=re.compile(r"viewuser.php\?uid=\d+"))
+        self.story.setMetadata('authorId',a['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/'+a['href'])
+        self.story.setMetadata('author',a.string)
+
+        # Find the chapters:
+        for chapter in soup.findAll('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"&chapter=\d+$")):
+            # just in case there's tags, like <i> in chapter titles.
+            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/'+chapter['href']+addurl))
+
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+
+        # <span class="label">Rated:</span> NC-17<br /> etc
+        labels = soup.findAll('span',{'class':'label'})
+        for labelspan in labels:
+            value = labelspan.nextSibling
+            label = labelspan.string
+
+            if 'Summary' in label:
+                ## Everything until the next span class='label'
+                svalue = ""
+                while not defaultGetattr(value,'class') == 'label':
+                    svalue += str(value)
+                    value = value.nextSibling
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))
+
+            if 'Rated' in label:
+                self.story.setMetadata('rating', value)
+
+            if 'Word count' in label:
+                self.story.setMetadata('numWords', value)
+
+            if 'Categories' in label:
+                cats = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=categories'))
+                catstext = [cat.string for cat in cats]
+                for cat in catstext:
+                    self.story.addToList('category',cat.string)
+
+            if 'Characters' in label:
+                chars = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=characters'))
+                charstext = [char.string for char in chars]
+                for char in charstext:
+                    self.story.addToList('characters',char.string)
+
+            ## Not all sites use Genre, but there's no harm to
+            ## leaving it in.  Check to make sure the type_id number
+            ## is correct, though--it's site specific.
+            if 'Genre' in label:
+                genres = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2')) # XXX
+                genrestext = [genre.string for genre in genres]
+                self.genre = ', '.join(genrestext)
+                for genre in genrestext:
+                    self.story.addToList('genre',genre.string)
+
+            ## Not all sites use Warnings, but there's no harm to
+            ## leaving it in.  Check to make sure the type_id number
+            ## is correct, though--it's site specific.
+            if 'Warnings' in label:
+                warnings = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2')) # XXX
+                warningstext = [warning.string for warning in warnings]
+                self.warning = ', '.join(warningstext)
+                for warning in warningstext:
+                    self.story.addToList('warnings',warning.string)
+
+            if 'Completed' in label:
+                if 'Yes' in value:
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+
+            if 'Published' in label:
+                self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
+            
+            if 'Updated' in label:
+                # there's a stray [ at the end.
+                #value = value[0:-1]
+                self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
+
+        try:
+            # Find Series name from series URL.
+            a = soup.find('a', href=re.compile(r"viewseries.php\?seriesid=\d+"))
+            series_name = a.string
+            series_url = 'http://'+self.host+'/'+a['href']
+
+            # use BeautifulSoup HTML parser to make everything easier to find.
+            seriessoup = bs.BeautifulSoup(self._fetchUrl(series_url))
+            storyas = seriessoup.findAll('a', href=re.compile(r'^viewstory.php\?sid=\d+$'))
+            i=1
+            for a in storyas:
+                if a['href'] == ('viewstory.php?sid='+self.story.getMetadata('storyId')):
+                    self.setSeries(series_name, i)
+                    break
+                i+=1
+            
+        except:
+            # I find it hard to care if the series parsing fails
+            pass
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        soup = bs.BeautifulStoneSoup(self._fetchUrl(url),
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+        
+        div = soup.find('div', {'id' : 'story'})
+
+        if None == div:
+            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,div)
@@ -0,0 +1,278 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+import cookielib as cl
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+# Search for XXX comments--that's where things are most likely to need changing.
+
+# This function is called by the downloader in all adapter_*.py files
+# in this dir to register the adapter class.  So it needs to be
+# updated to reflect the class below it.  That, plus getSiteDomain()
+# take care of 'Registering'.
+def getClass():
+    return PortkeyOrgAdapter # XXX
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class PortkeyOrgAdapter(BaseSiteAdapter): # XXX
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
+        self.password = ""
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.path.split('/',)[2])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        self._setURL('http://' + self.getSiteDomain() + '/story/'+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','prtky') # XXX
+
+        # If all stories from the site fall into the same category,
+        # the site itself isn't likely to label them as such, so we
+        # do.
+        self.story.addToList("category","Harry Potter") # XXX
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%d/%m/%y" # XXX
+            
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'fanfiction.portkey.org' # XXX
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/story/1234"
+
+    def getSiteURLPattern(self):
+        return re.escape("http://"+self.getSiteDomain()+"/story/")+r"\d+(/\d+)?$"
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        url = self.url
+        logging.debug("URL: "+url)
+
+        # portkey screws around with using a different URL to set the
+        # cookie and it's a pain.  So... cheat!
+        if self.is_adult or self.getConfig("is_adult"):
+            cookieproc = urllib2.HTTPCookieProcessor()
+            cookie = cl.Cookie(version=0, name='verify17', value='1',
+                               port=None, port_specified=False,
+                               domain=self.getSiteDomain(), domain_specified=False, domain_initial_dot=False,
+                               path='/', path_specified=True,
+                               secure=False,
+                               expires=time.time()+10000,
+                               discard=False,
+                               comment=None,
+                               comment_url=None,
+                               rest={'HttpOnly': None},
+                               rfc2109=False)
+            cookieproc.cookiejar.set_cookie(cookie)
+            self.opener = urllib2.build_opener(cookieproc)        
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        if "You must be over 18 years of age to view it" in data: # XXX 
+            raise exceptions.AdultCheckRequired(self.url)
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        #print data
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        # Find authorid and URL from... author url.
+        a = soup.find('a', href=re.compile(r"/profile/\d+"))
+        #print("======a:%s"%a)
+        self.story.setMetadata('authorId',a['href'].split('/')[-1])
+        self.story.setMetadata('authorUrl','http://'+self.host+a['href'])
+        self.story.setMetadata('author',a.string)
+
+        ## Going to get the rest from the author page.
+        authsoup = bs.BeautifulSoup(self._fetchUrl(self.story.getMetadata('authorUrl')))
+        
+        ## Title
+        titlea = authsoup.find('a', href=re.compile(r'/story/'+self.story.getMetadata('storyId')+"$"))
+        self.story.setMetadata('title',stripHTML(titlea))
+        metablock = titlea.parent
+        
+        # Find the chapters:
+        for chapter in soup.find('select',{'name':'select5'}).findAll('option', {'value':re.compile(r'/story/'+self.story.getMetadata('storyId')+"/\d+$")}):
+            # just in case there's tags, like <i> in chapter titles.
+            chtitle = stripHTML(chapter)
+            if not chtitle:
+                chtitle = "(Untitled Chapter)"
+            self.chapterUrls.append((chtitle,'http://'+self.host+chapter['value']))
+            
+        if len(self.chapterUrls) == 0:
+            self.chapterUrls.append((stripHTML(self.story.getMetadata('title')),url))
+
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+        # <SPAN class="dark-small-bold">Contents:</SPAN> <SPAN class="small-grey">NC17 </SPAN> 
+        # <SPAN class="dark-small-bold">Published: </SPAN><SPAN class="small-grey">12/11/07</SPAN>
+        # <SPAN class="dark-small-bold"><BR>
+        # Description:</SPAN> <SPAN class="small-black">A special book helps Harry tap into the power the Dark Lord knows not.  Of course it’s a book on sex magic and rituals… but Harry’s not complaining.  Spurned on by the ghost of a pervert founder, Harry leads his friends in the hunt for Voldemort’s Horcruxes.
+        # EROTIC COMEDY!  Loads of crude humor and sexual situations!
+        # </SPAN>
+        labels = metablock.findAll('span',{'class':'dark-small-bold'})
+        for labelspan in labels:
+            value = labelspan.findNext('span').string
+            label = stripHTML(labelspan)
+#            print("\nlabel:%s\nlabel:%s\nvalue:%s\n"%(labelspan,label,value))
+
+            if 'Description' in label:
+                self.setDescription(url,value)
+
+            if 'Contents' in label:
+                self.story.setMetadata('rating', value)
+
+            if 'Words' in label:
+                self.story.setMetadata('numWords', value)
+
+            # if 'Categories' in label:
+            #     cats = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=categories'))
+            #     catstext = [cat.string for cat in cats]
+            #     for cat in catstext:
+            #         self.story.addToList('category',cat.string)
+
+            # if 'Characters' in label:
+            #     chars = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=characters'))
+            #     charstext = [char.string for char in chars]
+            #     for char in charstext:
+            #         self.story.addToList('characters',char.string)
+
+            if 'Genre' in label:
+                # genre is typo'ed on the site--it falls between the
+                # dark-small-bold label and dark-small-bold content
+                # spans.
+                svalue = ""
+                value = labelspan.nextSibling
+                while not defaultGetattr(value,'class') == 'dark-small-bold':
+                    svalue += str(value)
+                    value = value.nextSibling
+                    
+                for genre in svalue.split("/"):
+                    genre = genre.strip()
+                    if genre != 'None':
+                        self.story.addToList('genre',genre)
+
+            ## Not all sites use Warnings, but there's no harm to
+            ## leaving it in.  Check to make sure the type_id number
+            ## is correct, though--it's site specific.
+            # if 'Warnings' in label:
+            #     warnings = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2')) # XXX
+            #     warningstext = [warning.string for warning in warnings]
+            #     self.warning = ', '.join(warningstext)
+            #     for warning in warningstext:
+            #         self.story.addToList('warnings',warning.string)
+
+            if 'Status' in label:
+                if 'Completed' in value:
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+
+            if 'Published' in label:
+                self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
+            
+            if 'Updated' in label:
+                self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
+
+        # try:
+        #     # Find Series name from series URL.
+        #     a = metablock.find('a', href=re.compile(r"viewseries.php\?seriesid=\d+"))
+        #     series_name = a.string
+        #     series_url = 'http://'+self.host+'/'+a['href']
+
+        #     # use BeautifulSoup HTML parser to make everything easier to find.
+        #     seriessoup = bs.BeautifulSoup(self._fetchUrl(series_url))
+        #     storyas = seriessoup.findAll('a', href=re.compile(r'^viewstory.php\?sid=\d+$'))
+        #     i=1
+        #     for a in storyas:
+        #         if a['href'] == ('viewstory.php?sid='+self.story.getMetadata('storyId')):
+        #             self.setSeries(series_name, i)
+        #             break
+        #         i+=1            
+        # except:
+        #     # I find it hard to care if the series parsing fails
+        #     pass
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+        data = self._fetchUrl(url)
+        soup = bs.BeautifulStoneSoup(data,
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+
+        #print("soup:%s"%soup)
+        tag = soup.find('td', {'class' : 'story'})
+        if tag == None and "<center><b>Chapter does not exist!</b></center>" in data:
+            print("Chapter is missing at: %s"%url)
+            return  self.utf8FromSoup(url,bs.BeautifulStoneSoup("<div><p><center><b>Chapter does not exist!</b></center></p><p>Chapter is missing at: <a href='%s'>%s</a></p></div>"%(url,url)))
+        tag.name='div' # force to be a div to avoid problems with nook.
+
+        centers = tag.findAll('center')
+        # first two and last two center tags are some script, 'report
+        # story', 'report story' and an ad.
+        centers[0].extract()
+        centers[1].extract()
+        centers[-1].extract()
+        centers[-2].extract()
+
+        if None == tag:
+            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,tag)
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class PotionsAndSnitchesNetSiteAdapter(BaseSiteAdapter):

@@ -61,14 +61,6 @@ class PotionsAndSnitchesNetSiteAdapter(BaseSiteAdapter):
    def getSiteURLPattern(self):
        return re.escape("http://")+r"(www\.)?"+re.escape("potionsandsnitches.net/fanfiction/viewstory.php?sid=")+r"\d+$"

-    def needToLoginCheck(self, data):
-        if 'Registered Users Only' in data \
-                or 'There is no such account on our website' in data \
-                or "That password doesn't match the one in our database" in data:
-          return True
-        else:
-          return False
-
    def extractChapterUrlsAndMetadata(self):

        url = self.url+'&index=1'
@@ -131,7 +123,8 @@ class PotionsAndSnitchesNetSiteAdapter(BaseSiteAdapter):
                while not defaultGetattr(value,'class') == 'listbox':
                    svalue += str(value)
                    value = value.nextSibling
-                self.story.setMetadata('description',stripHTML(svalue))
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))

            if 'Rated' in label:
                self.story.setMetadata('rating', value)
@@ -209,7 +202,7 @@ class PotionsAndSnitchesNetSiteAdapter(BaseSiteAdapter):
        if None == div:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(div)
+        return self.utf8FromSoup(url,div)

 def getClass():
    return PotionsAndSnitchesNetSiteAdapter
@@ -24,7 +24,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 # This function is called by the downloader in all adapter_*.py files
 # in this dir to register the adapter class.  So it needs to be
@@ -45,9 +45,6 @@ class SiyeCoUkAdapter(BaseSiteAdapter): # XXX
                               # Most sites that claim to be
                               # iso-8859-1 (and some that claim to be
                               # utf8) are really windows-1252.
-        # self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
-        # self.password = ""
-        # self.is_adult=False
        
        # get storyId from url--url validation guarantees query is only sid=1234
        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
@@ -83,53 +80,9 @@ class SiyeCoUkAdapter(BaseSiteAdapter): # XXX
    def getSiteURLPattern(self):
        return re.escape("http://")+r"(www\.)?siye\.co\.uk/(siye/)?"+re.escape("viewstory.php?sid=")+r"\d+$"

-    # ## Login seems to be reasonably standard across eFiction sites.
-    # def needToLoginCheck(self, data):
-    #     if 'Registered Users Only' in data \
-    #             or 'There is no such account on our website' in data \
-    #             or "That password doesn't match the one in our database" in data:
-    #         return True
-    #     else:
-    #         return False
-        
-    # def performLogin(self, url):
-    #     params = {}
-
-    #     if self.password:
-    #         params['penname'] = self.username
-    #         params['password'] = self.password
-    #     else:
-    #         params['penname'] = self.getConfig("username")
-    #         params['password'] = self.getConfig("password")
-    #     params['cookiecheck'] = '1'
-    #     params['submit'] = 'Submit'
-    
-    #     loginUrl = 'http://' + self.getSiteDomain() + '/user.php?action=login'
-    #     logging.debug("Will now login to URL (%s) as (%s)" % (loginUrl,
-    #                                                           params['penname']))
-    
-    #     d = self._fetchUrl(loginUrl, params)
-    
-    #     if "Member Account" not in d : #Member Account
-    #         logging.info("Failed to login to URL %s as %s" % (loginUrl,
-    #                                                           params['penname']))
-    #         raise exceptions.FailedToLogin(url,params['penname'])
-    #         return False
-    #     else:
-    #         return True
-
    ## Getting the chapter list and the meta data, plus 'is adult' checking.
    def extractChapterUrlsAndMetadata(self):

-        # if self.is_adult or self.getConfig("is_adult"):
-        #     # Weirdly, different sites use different warning numbers.
-        #     # If the title search below fails, there's a good chance
-        #     # you need a different number.  print data at that point
-        #     # and see what the 'click here to continue' url says.
-        #     addurl = "&ageconsent=ok&warning=4" # XXX
-        # else:
-        #     addurl=""
-
        # index=1 makes sure we see the story chapter index.  Some
        # sites skip that for one-chapter stories.
        # Except it doesn't this time. :-/
@@ -144,20 +97,6 @@ class SiyeCoUkAdapter(BaseSiteAdapter): # XXX
            else:
                raise e

-        # if self.needToLoginCheck(data):
-        #     # need to log in for this one.
-        #     self.performLogin(url)
-        #     data = self._fetchUrl(url)
-
-        # # The actual text that is used to announce you need to be an
-        # # adult varies from site to site.  Again, print data before
-        # # the title search to troubleshoot.
-        # if "Age Consent Required" in data: # XXX 
-        #     raise exceptions.AdultCheckRequired(self.url)
-            
-        # if "Access denied. This story has not been validated by the adminstrators of this site." in data:
-        #     raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")
-            
        # use BeautifulSoup HTML parser to make everything easier to find.
        soup = bs.BeautifulSoup(data)
        # print data
@@ -227,7 +166,8 @@ class SiyeCoUkAdapter(BaseSiteAdapter): # XXX

            if part.startswith("Summary:"):
                part = part[part.find(':')+1:]
-                self.story.setMetadata('description',part)
+                self.setDescription(url,part)
+                #self.story.setMetadata('description',part)
                
        # want to get the next tr of the table.
        #print("%s"%titlea.parent.parent.findNextSibling('tr'))
@@ -295,4 +235,4 @@ class SiyeCoUkAdapter(BaseSiteAdapter): # XXX
        if None == story:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(story)
+        return self.utf8FromSoup(url,story)
@@ -0,0 +1,234 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+
+def getClass():
+    return SquidgeOrgPejaAdapter
+
+## XXX IMPORTANT NOTE!!  This adapter is for squidge.org/peja ONLY!
+## There are lots of other sites and stuff under squidge.org that
+## we're not supporting.  If/When we ever want to support more
+## sections of squidge.org, FFDL will need to be changed more
+## fundamentally to find different adapters under the same domain.
+##
+## For now, I've only implemented the part for ini section names so
+## if/when more adapters under squidge.org come along, existing ini
+## files will still work correctly.
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class SquidgeOrgPejaAdapter(BaseSiteAdapter):
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["utf8",
+                       "Windows-1252"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
+        self.password = ""
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        self._setURL('http://' + self.getSiteDomain() + '/peja/cgi-bin/viewstory.php?sid='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','wwomb')
+        
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%m/%d/%y"
+            
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'www.squidge.org'
+    
+    @classmethod # must be @staticmethod, don't remove it.
+    def getConfigSection(cls):
+        # The config section name.  Only override if != site domain.
+        return cls.getSiteDomain()+'/peja'
+    
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/peja/cgi-bin/viewstory.php?sid=1234"
+
+    def getSiteURLPattern(self):
+        return re.escape("http://"+self.getSiteDomain()+"/peja/cgi-bin/viewstory.php?sid=")+r"\d+$"        
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        url = self.url
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        data = self._fetchUrl(url)
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        # print data
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        # Find authorid and URL from... author url.
+        author = soup.find('div', {'id':"pagetitle"}).find('a')
+        self.story.setMetadata('authorId',author['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/peja/cgi-bin/'+author['href'])
+        self.story.setMetadata('author',author.string)
+		
+        authorSoup = bs.BeautifulSoup(self._fetchUrl(self.story.getMetadata('authorUrl')))
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        title = authorSoup.find('a',{'href':'viewstory.php?sid='+self.story.getMetadata('storyId')})
+        self.story.setMetadata('title',title.string)
+        titleblock=title.parent.parent
+        
+        chapterselect=soup.find('select',{'name':'chapter'})
+        if chapterselect:
+            for ch in chapterselect.findAll('option'):
+                self.chapterUrls.append((stripHTML(ch),'http://'+self.host+'/peja/cgi-bin/viewstory.php?sid='+self.story.getMetadata('storyId')+'&chapter='+ch['value']))
+        else:
+            self.chapterUrls.append((title,url))
+		
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+        
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+
+        # <span class="classification">Rated:</span> NC-17<br /> etc
+        labels = titleblock.findAll('span',{'class':'classification'})
+        for labelspan in labels:
+            value = labelspan.nextSibling
+            label = labelspan.string
+
+            if 'Summary' in label:
+                ## Everything until the next span class='label'
+                svalue = ""
+                while not defaultGetattr(value,'class') == 'classification':
+                    svalue += str(value)
+                    value = value.nextSibling
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))
+
+            if 'Rated' in label:
+                self.story.setMetadata('rating', value)
+
+            if 'Characters' in label:
+                for char in value.split(','):
+                    self.story.addToList('characters',char.strip())
+
+            if 'Genre' in label:
+                for genre in value.split(','):
+                    if genre.strip() != "None":
+                        self.story.addToList('genre',genre.strip())
+
+            if 'Warnings' in label:
+                for warning in value.split(','):
+                    if warning.strip() != 'None':
+                        self.story.addToList('warnings',warning.strip())
+
+            if 'Word count' in label:
+                self.story.setMetadata('numWords', value)
+
+            if 'Fandoms' in label:
+                cats = labelspan.parent.findAll('a',href=re.compile(r'categories.php'))
+                catstext = [cat.string for cat in cats]
+                for cat in catstext:
+                    self.story.addToList('category',cat.string)
+
+            if 'Completed' in label:
+                if 'Yes' in value:
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+
+            if 'Published' in label:
+                self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
+            
+            if 'Updated' in label:
+                # there's a stray [ at the end.
+                #value = value[0:-1]
+                self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
+
+        try:
+            # Find Series name from series URL.
+            # http://www.squidge.org/peja/cgi-bin/series.php?seriesid=254
+            a = titleblock.find('a', href=re.compile(r"series.php\?seriesid=\d+"))
+            series_name = a.string
+            series_url = 'http://'+self.host+'/peja/cgi-bin/'+a['href']
+    
+            # use BeautifulSoup HTML parser to make everything easier to find.
+            seriessoup = bs.BeautifulSoup(self._fetchUrl(series_url))
+            storyas = seriessoup.findAll('a', href=re.compile(r'^viewstory.php\?sid=\d+$'))
+            i=1
+            for a in storyas:
+                if a['href'] == ('viewstory.php?sid='+self.story.getMetadata('storyId')):
+                    self.setSeries(series_name, i)
+                    break
+                i+=1
+                
+        except:
+            # I find it hard to care if the series parsing fails
+            pass
+            
+	
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        soup = bs.BeautifulStoneSoup(self._fetchUrl(url),
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+        
+        chaptext = soup.find('div',{'id':"story"}).find('span')
+
+        if None == chaptext:
+            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,chaptext)
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class TenhawkPresentsComSiteAdapter(BaseSiteAdapter):

@@ -164,7 +164,8 @@ class TenhawkPresentsComSiteAdapter(BaseSiteAdapter):
                while not defaultGetattr(value,'class') == 'label':
                    svalue += str(value)
                    value = value.nextSibling
-                self.story.setMetadata('description',stripHTML(svalue))
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))

            if 'Rated' in label:
                self.story.setMetadata('rating', value)
@@ -238,7 +239,7 @@ class TenhawkPresentsComSiteAdapter(BaseSiteAdapter):
        if None == span:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(span)
+        return self.utf8FromSoup(url,span)

 def getClass():
    return TenhawkPresentsComSiteAdapter
@@ -22,7 +22,7 @@ import logging
 from .. import BeautifulSoup as bs
 from .. import exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class TestSiteAdapter(BaseSiteAdapter):

@@ -78,7 +78,6 @@ class TestSiteAdapter(BaseSiteAdapter):
 Some more longer description.  "I suck at summaries!"  "Better than it sounds!"  "My first fic"
 ''')
        self.story.setMetadata('datePublished',makeDate("1975-03-15","%Y-%m-%d"))
-        self.story.setMetadata('dateCreated',datetime.datetime.now())
        if self.story.getMetadata('storyId') == '669':
            self.story.setMetadata('dateUpdated',datetime.datetime.now())
        else:
@@ -127,7 +126,7 @@ Some more longer description.  "I suck at summaries!"  "Better than it sounds!"
                            ('Chapter 4',self.url+"&chapter=5"),
                            ('Chapter 5',self.url+"&chapter=6"),
                            ('Chapter 6',self.url+"&chapter=6"),
-                            ('Chapter 7',self.url+"&chapter=6"),
+                            # ('Chapter 7',self.url+"&chapter=6"),
                            # ('Chapter 8',self.url+"&chapter=6"),
                            # ('Chapter 9',self.url+"&chapter=6"),
                            # ('Chapter 0',self.url+"&chapter=6"),
@@ -178,20 +177,23 @@ Some more longer description.  "I suck at summaries!"  "Better than it sounds!"
        else:
            text=u'''
 <div>
-<h3>Chapter</h3>
+<h3>Chapter title from site</h3>
 <p><center>Centered text</center></p>
-<p>Lorem '''+self.crazystring+''' <i>italics</i>, <b>bold</b>, <u>underline</u> consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
+<p>Lorem '''+self.crazystring+u''' <i>italics</i>, <b>bold</b>, <u>underline</u> consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
 br breaks<br><br>
+Puella Magi Madoka Magica/魔法少女まどか★マギカ
+<a href="http://code.google.com/p/fanficdownloader/wiki/FanFictionDownLoaderPluginWithReadingList" title="Tilt-a-Whirl by Jim &amp; Sarah, on Flickr"><img src="http://i.imgur.com/bo8eD.png"></a><br/>
 br breaks<br><br>
+Don't&#8212e;ver&#8212d;o&#8212;that&#8212a;gain, &#27861; &#xE9;
 <hr>
 horizontal rules
-<hr>
+<hr size=1 noshade>
 <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
 <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
 </div>
 '''
        soup = bs.BeautifulStoneSoup(text,selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
-        return utf8FromSoup(soup)
+        return self.utf8FromSoup(url,soup)

 def getClass():
    return TestSiteAdapter
@@ -0,0 +1,293 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+# This function is called by the downloader in all adapter_*.py files
+# in this dir to register the adapter class.  So it needs to be
+# updated to reflect the class below it.  That, plus getSiteDomain()
+# take care of 'Registering'.
+def getClass():
+    return TheQuidditchPitchOrgAdapter # XXX
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class TheQuidditchPitchOrgAdapter(BaseSiteAdapter): # XXX
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
+        self.password = ""
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+        # normalized story URL.
+        # XXX Most sites don't have the  part.  Replace all to remove it usually.
+        self._setURL('http://' + self.getSiteDomain() + '/viewstory.php?sid='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','tqdpch') # XXX
+
+        # If all stories from the site fall into the same category,
+        # the site itself isn't likely to label them as such, so we
+        # do.
+        self.story.addToList("category","Harry Potter") # XXX
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%m/%d/%Y" # XXX
+            
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'thequidditchpitch.org' # XXX
+
+    @classmethod
+    def getAcceptDomains(cls):
+        return ['www.thequidditchpitch.org','thequidditchpitch.org']
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/viewstory.php?sid=1234"
+
+    def getSiteURLPattern(self):
+        return re.escape("http://")+"(www\.)?"+re.escape(self.getSiteDomain()+"/viewstory.php?sid=")+r"\d+$"
+
+    ## Login seems to be reasonably standard across eFiction sites.
+    def needToLoginCheck(self, data):
+        if 'Registered Users Only - Not suitable for readers under the age of legal consent in their country.' in data \
+                or 'There is no such account on our website' in data \
+                or "That password doesn't match the one in our database" in data:
+            return True
+        else:
+            return False
+        
+    def performLogin(self, url):
+        params = {}
+
+        if self.password:
+            params['penname'] = self.username
+            params['password'] = self.password
+        else:
+            params['penname'] = self.getConfig("username")
+            params['password'] = self.getConfig("password")
+        params['cookiecheck'] = '1'
+        params['submit'] = 'Submit'
+    
+        loginUrl = 'http://' + self.getSiteDomain() + '/user.php?action=login'
+        logging.debug("Will now login to URL (%s) as (%s)" % (loginUrl,
+                                                              params['penname']))
+    
+        d = self._fetchUrl(loginUrl, params)
+    
+        if "Member Account" not in d : #Member Account
+            logging.info("Failed to login to URL %s as %s" % (loginUrl,
+                                                              params['penname']))
+            raise exceptions.FailedToLogin(url,params['penname'])
+            return False
+        else:
+            return True
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        if self.is_adult or self.getConfig("is_adult"):
+            # Weirdly, different sites use different warning numbers.
+            # If the title search below fails, there's a good chance
+            # you need a different number.  print data at that point
+            # and see what the 'click here to continue' url says.
+            addurl = "&ageconsent=ok&warning=4" # XXX
+        else:
+            addurl=""
+
+        # index=1 makes sure we see the story chapter index.  Some
+        # sites skip that for one-chapter stories.
+        url = self.url+'&index=1'+addurl
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+
+        if self.needToLoginCheck(data):
+            # need to log in for this one.
+            self.performLogin(url)
+            data = self._fetchUrl(url)
+
+        # The actual text that is used to announce you need to be an
+        # adult varies from site to site.  Again, print data before
+        # the title search to troubleshoot.
+        if ("Not suitable for readers under the age of legal consent in their country." in data \
+                or "Not suitable for readers under 16 yrs. \r\nStories may contain violence, slight nudity, and/or sexual situations." in data ) \
+                and not (self.is_adult or self.getConfig("is_adult")): # XXX 
+            raise exceptions.AdultCheckRequired(self.url)
+            
+        if "Access denied. This story has not been validated by the adminstrators of this site." in data:
+            raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        #print data
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        ## Title
+        a = soup.find('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')))
+        self.story.setMetadata('title',a.string)
+        
+        # Find authorid and URL from... author url.
+        a = soup.find('a', href=re.compile(r"viewuser.php\?uid=\d+"))
+        self.story.setMetadata('authorId',a['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/'+a['href'])
+        self.story.setMetadata('author',a.string)
+
+        # Find the chapters:
+        for chapter in soup.findAll('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"&chapter=\d+$")):
+            # just in case there's tags, like <i> in chapter titles.
+            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/'+chapter['href']+addurl))
+
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+
+        # <span class="label">Rated:</span> NC-17<br /> etc
+        labels = soup.findAll('span',{'class':'label'})
+        for labelspan in labels:
+            value = labelspan.nextSibling
+            label = labelspan.string
+
+            if 'Summary' in label:
+                ## Everything until the next span class='label'
+                svalue = ""
+                while not defaultGetattr(value,'class') == 'label':
+                    svalue += str(value)
+                    value = value.nextSibling
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))
+
+            if 'Rated' in label:
+                self.story.setMetadata('rating', value)
+
+            if 'Word count' in label:
+                self.story.setMetadata('numWords', value)
+
+            if 'Categories' in label:
+                cats = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=categories'))
+                catstext = [cat.string for cat in cats]
+                for cat in catstext:
+                    self.story.addToList('category',cat.string)
+
+            if 'Characters' in label:
+                chars = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=characters'))
+                charstext = [char.string for char in chars]
+                for char in charstext:
+                    self.story.addToList('characters',char.string)
+
+            if 'Genre' in label:
+                genres = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=1')) # XXX
+                genrestext = [genre.string for genre in genres]
+                self.genre = ', '.join(genrestext)
+                for genre in genrestext:
+                    self.story.addToList('genre',genre.string)
+
+            if 'Warnings' in label:
+                warnings = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2')) # XXX
+                warningstext = [warning.string for warning in warnings]
+                self.warning = ', '.join(warningstext)
+                for warning in warningstext:
+                    self.story.addToList('warnings',warning.string)
+
+            if 'Completed' in label:
+                if 'Yes' in value:
+                    self.story.setMetadata('status', 'Completed')
+                else:
+                    self.story.setMetadata('status', 'In-Progress')
+
+            if 'Published' in label:
+                self.story.setMetadata('datePublished', makeDate(stripHTML(value), self.dateformat))
+            
+            if 'Updated' in label:
+                # there's a stray [ at the end.
+                #value = value[0:-1]
+                self.story.setMetadata('dateUpdated', makeDate(stripHTML(value), self.dateformat))
+
+        try:
+            # Find Series name from series URL.
+            a = soup.find('a', href=re.compile(r"viewseries.php\?seriesid=\d+"))
+            series_name = a.string
+            series_url = 'http://'+self.host+'/'+a['href']
+
+            # use BeautifulSoup HTML parser to make everything easier to find.
+            seriessoup = bs.BeautifulSoup(self._fetchUrl(series_url))
+            storyas = seriessoup.findAll('a', href=re.compile(r'^viewstory.php\?sid=\d+$'))
+            i=1
+            for a in storyas:
+                if a['href'] == ('viewstory.php?sid='+self.story.getMetadata('storyId')):
+                    self.setSeries(series_name, i)
+                    break
+                i+=1
+            
+        except:
+            # I find it hard to care if the series parsing fails
+            pass
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        
+        soup = bs.BeautifulStoneSoup(self._fetchUrl(url),
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+
+        # span?  Really?  span?  Yeah... I don't think so.
+        div = soup.find('span', {'style' : 'font-size: 100%;'})
+        div.name='div'
+
+        if None == div:
+            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,div)
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class TheWritersCoffeeShopComSiteAdapter(BaseSiteAdapter):

@@ -166,7 +166,7 @@ class TheWritersCoffeeShopComSiteAdapter(BaseSiteAdapter):
                while not defaultGetattr(value,'class') == 'label':
                    svalue += str(value)
                    value = value.nextSibling
-                self.story.setMetadata('description',stripHTML(svalue))
+                self.setDescription(url,svalue)

            if 'Rated' in label:
                self.story.setMetadata('rating', value)
@@ -245,7 +245,7 @@ class TheWritersCoffeeShopComSiteAdapter(BaseSiteAdapter):
        if None == span:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(span)
+        return self.utf8FromSoup(url,span)

 def getClass():
    return TheWritersCoffeeShopComSiteAdapter
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class TwistingTheHellmouthSiteAdapter(BaseSiteAdapter):

@@ -127,6 +127,8 @@ class TwistingTheHellmouthSiteAdapter(BaseSiteAdapter):
            else:
                raise e
            
+        descurl = url
+            
        if "<h2>Story Not Found</h2>" in data:
            raise exceptions.StoryDoesNotExist(url)
        
@@ -154,12 +156,14 @@ class TwistingTheHellmouthSiteAdapter(BaseSiteAdapter):
            # going to pull part of the meta data from author list page.
            logging.debug("**AUTHOR** URL: "+self.story.getMetadata('authorUrl'))
            authordata = self._fetchUrl(self.story.getMetadata('authorUrl'))
+            descurl=self.story.getMetadata('authorUrl')
            authorsoup = bs.BeautifulSoup(authordata)
            # author can have several pages, scan until we find it.
            while( not authorsoup.find('a', href=re.compile(r"^/Story-"+self.story.getMetadata('storyId'))) ):
                nextpage = 'http://'+self.host+authorsoup.find('a', {'class':'arrowf'})['href']
                logging.debug("**AUTHOR** nextpage URL: "+nextpage)
                authordata = self._fetchUrl(nextpage)
+                descurl=nextpage
                authorsoup = bs.BeautifulSoup(authordata)
        except urllib2.HTTPError, e:
            if e.code == 404:
@@ -168,20 +172,22 @@ class TwistingTheHellmouthSiteAdapter(BaseSiteAdapter):
                raise e

        storydiv = authorsoup.find('div', {'id':'st'+self.story.getMetadata('storyId'), 'class':re.compile(r"storylistitem")})
-        self.story.setMetadata('description',stripHTML(storydiv.find('div',{'class':'storydesc'})))
+        self.setDescription(descurl,storydiv.find('div',{'class':'storydesc'}))
+        #self.story.setMetadata('description',stripHTML(storydiv.find('div',{'class':'storydesc'})))
        self.story.setMetadata('title',stripHTML(storydiv.find('a',{'class':'storylink'})))

        verticaltable = soup.find('table', {'class':'verticaltable'})

        BtVS = True
+        BtVSNonX = False
        for cat in verticaltable.findAll('a', href=re.compile(r"^/Category-")):
            if cat.string not in ['General', 'Non-BtVS/AtS Stories', 'BtVS/AtS Non-Crossover', 'Non-BtVS Crossovers']:
                self.story.addToList('category',cat.string)
            else:
                if 'Non-BtVS' in cat.string:
                    BtVS = False
-        if BtVS:
-            self.story.addToList('category','Buffy: The Vampire Slayer')
+                if 'BtVS/AtS Non-Crossover' == cat.string:
+                    BtVSNonX = True

        verticaltabletds = verticaltable.findAll('td')
        self.story.setMetadata('rating', verticaltabletds[2].string)
@@ -199,6 +205,13 @@ class TwistingTheHellmouthSiteAdapter(BaseSiteAdapter):
        for icon in storydiv.find('span',{'class':'storyicons'}).findAll('img'):
            if( icon['title'] not in ['Non-Crossover'] ) :
                self.story.addToList('genre',icon['title'])
+            else:
+                if not BtVSNonX:
+                    BtVS = False # Don't add BtVS if Non-Crossover, unless it's a BtVS/AtS Non-Crossover
+
+        print("BtVS: %s BtVSNonX: %s"%(BtVS,BtVSNonX))
+        if BtVS:
+            self.story.addToList('category','Buffy: The Vampire Slayer')
            
        # Find the chapter selector 
        select = soup.find('select', { 'name' : 'chapnav' } )
@@ -238,7 +251,7 @@ class TwistingTheHellmouthSiteAdapter(BaseSiteAdapter):
            div.find('h3').extract()
        except:
            pass
-        return utf8FromSoup(div)
+        return self.utf8FromSoup(url,div)

 def getClass():
    return TwistingTheHellmouthSiteAdapter
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class TwilightedNetSiteAdapter(BaseSiteAdapter):

@@ -162,7 +162,7 @@ class TwilightedNetSiteAdapter(BaseSiteAdapter):
                while not defaultGetattr(value,'class') == 'label':
                    svalue += str(value)
                    value = value.nextSibling
-                self.story.setMetadata('description',stripHTML(svalue))
+                self.setDescription(url,svalue)

            if 'Rated' in label:
                self.story.setMetadata('rating', value)
@@ -243,7 +243,7 @@ class TwilightedNetSiteAdapter(BaseSiteAdapter):
        if None == span:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(span)
+        return self.utf8FromSoup(url,span)

 def getClass():
    return TwilightedNetSiteAdapter
@@ -25,7 +25,7 @@ from .. import BeautifulSoup as bs
 from ..htmlcleanup import stripHTML
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class TwiwriteNetSiteAdapter(BaseSiteAdapter):

@@ -38,6 +38,7 @@ class TwiwriteNetSiteAdapter(BaseSiteAdapter):
                               # iso-8859-1 (and some that claim to be
                               # utf8) are really windows-1252.
        self.story.addToList("category","Twilight")
+        self.is_adult = False
        self.username = "NoneGiven" # if left empty, twiwrite.net doesn't return any message at all.
        self.password = ""
        
@@ -99,7 +100,16 @@ class TwiwriteNetSiteAdapter(BaseSiteAdapter):

    def extractChapterUrlsAndMetadata(self):

-        url = self.url+'&index=1'
+        if self.is_adult or self.getConfig("is_adult"):
+            # Weirdly, different sites use different warning numbers.
+            # If the title search below fails, there's a good chance
+            # you need a different number.  print data at that point
+            # and see what the 'click here to continue' url says.
+            addurl = "&ageconsent=ok&warning=1" # XXX
+        else:
+            addurl=""
+            
+        url = self.url+'&index=1'+addurl
        logging.debug("URL: "+url)

        try:
@@ -118,6 +128,9 @@ class TwiwriteNetSiteAdapter(BaseSiteAdapter):
        if "Access denied. This story has not been validated by the adminstrators of this site." in data:
            raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")

+        if "Contains Explicit Content for mature adults only! May contain graphic violence, mature sexual situations, and explicit language. Read with caution." in data:
+            raise exceptions.AdultCheckRequired(self.url)
+        
        # problems with some stories, but only in calibre.  I suspect
        # issues with different SGML parsers in python.  This is a
        # nasty hack, but it works.
@@ -139,7 +152,7 @@ class TwiwriteNetSiteAdapter(BaseSiteAdapter):
        # Find the chapters:
        for chapter in soup.findAll('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"&chapter=\d+$")):
            # just in case there's tags, like <i> in chapter titles.
-            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/'+chapter['href']))
+            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/'+chapter['href']+addurl))

        self.story.setMetadata('numChapters',len(self.chapterUrls))

@@ -169,7 +182,8 @@ class TwiwriteNetSiteAdapter(BaseSiteAdapter):
                while not defaultGetattr(value,'class') == 'label':
                    svalue += str(value)
                    value = value.nextSibling
-                self.story.setMetadata('description',stripHTML(svalue))
+                self.setDescription(url,svalue)
+                #self.story.setMetadata('description',stripHTML(svalue))

            if 'Rated' in label:
                self.story.setMetadata('rating', value)
@@ -255,7 +269,7 @@ class TwiwriteNetSiteAdapter(BaseSiteAdapter):
        if None == span:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(span)
+        return self.utf8FromSoup(url,span)

 def getClass():
    return TwiwriteNetSiteAdapter
@@ -23,7 +23,7 @@ import urllib2
 from .. import BeautifulSoup as bs
 from .. import exceptions as exceptions

-from base_adapter import BaseSiteAdapter, utf8FromSoup, makeDate
+from base_adapter import BaseSiteAdapter,  makeDate

 class WhoficComSiteAdapter(BaseSiteAdapter):

@@ -120,9 +120,10 @@ class WhoficComSiteAdapter(BaseSiteAdapter):
        # link instead to find the appropriate metadata.
        a = soup.find('a', href=re.compile(r'reviews.php\?sid='+self.story.getMetadata('storyId')))
        metadata = a.findParent('td')
-        metadatachunks = utf8FromSoup(metadata).split('<br />')
+        metadatachunks = self.utf8FromSoup(None,metadata).split('<br />')
        # process metadata for this story.
-        self.story.setMetadata('description', metadatachunks[1])
+        self.setDescription(url,metadatachunks[1])
+        #self.story.setMetadata('description', metadatachunks[1])

        # First line of the stuff with ' - ' separators
        moremeta = metadatachunks[2]
@@ -224,7 +225,7 @@ class WhoficComSiteAdapter(BaseSiteAdapter):
        if None == span:
            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
    
-        return utf8FromSoup(span)
+        return self.utf8FromSoup(url,span)

 def getClass():
    return WhoficComSiteAdapter
@@ -0,0 +1,227 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2011 Fanficdownloader team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import time
+import logging
+import re
+import urllib2
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML
+from .. import exceptions as exceptions
+
+from base_adapter import BaseSiteAdapter,  makeDate
+
+
+def getClass():
+    return WraithBaitComAdapter
+
+# Class name has to be unique.  Our convention is camel case the
+# sitename with Adapter at the end.  www is skipped.
+class WraithBaitComAdapter(BaseSiteAdapter):
+
+    def __init__(self, config, url):
+        BaseSiteAdapter.__init__(self, config, url)
+
+        self.decode = ["Windows-1252",
+                       "utf8"] # 1252 is a superset of iso-8859-1.
+                               # Most sites that claim to be
+                               # iso-8859-1 (and some that claim to be
+                               # utf8) are really windows-1252.
+        self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
+        self.password = ""
+        self.is_adult=False
+        
+        # get storyId from url--url validation guarantees query is only sid=1234
+        self.story.setMetadata('storyId',self.parsedUrl.query.split('=',)[1])
+        logging.debug("storyId: (%s)"%self.story.getMetadata('storyId'))
+        
+
+        self._setURL('http://' + self.getSiteDomain() + '/viewstory.php?sid='+self.story.getMetadata('storyId'))
+        
+        # Each adapter needs to have a unique site abbreviation.
+        self.story.setMetadata('siteabbrev','wb')
+
+        # If all stories from the site fall into the same category,
+        # the site itself isn't likely to label them as such, so we
+        # do.
+        self.story.addToList("category","Stargate")
+
+        # The date format will vary from site to site.
+        # http://docs.python.org/library/datetime.html#strftime-strptime-behavior
+        self.dateformat = "%d  %b %Y"
+            
+    @staticmethod # must be @staticmethod, don't remove it.
+    def getSiteDomain():
+        # The site domain.  Does have www here, if it uses it.
+        return 'www.wraithbait.com'
+
+    def getSiteExampleURLs(self):
+        return "http://"+self.getSiteDomain()+"/viewstory.php?sid=1234"
+
+    def getSiteURLPattern(self):
+        return re.escape("http://"+self.getSiteDomain()+"/viewstory.php?sid=")+r"\d+$"
+
+    ## Getting the chapter list and the meta data, plus 'is adult' checking.
+    def extractChapterUrlsAndMetadata(self):
+
+        if self.is_adult or self.getConfig("is_adult"):
+            # Weirdly, different sites use different warning numbers.
+            # If the title search below fails, there's a good chance
+            # you need a different number.  print data at that point
+            # and see what the 'click here to continue' url says.
+            addurl = "&ageconsent=ok&warning=12"
+        else:
+            addurl=""
+
+        # index=1 makes sure we see the story chapter index.  Some
+        # sites skip that for one-chapter stories.
+        url = self.url+'&index=1'+addurl
+        logging.debug("URL: "+url)
+
+        try:
+            data = self._fetchUrl(url)
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                raise exceptions.StoryDoesNotExist(self.url)
+            else:
+                raise e
+				
+        if "for adults only" in data:
+            raise exceptions.AdultCheckRequired(self.url)
+            
+        if "Access denied. This story has not been validated by the adminstrators of this site." in data:
+            raise exceptions.FailedToDownload(self.getSiteDomain() +" says: Access denied. This story has not been validated by the adminstrators of this site.")
+            
+        # use BeautifulSoup HTML parser to make everything easier to find.
+        soup = bs.BeautifulSoup(data)
+        # print data
+
+        # Now go hunting for all the meta data and the chapter list.
+        
+        ## Title
+        pt = soup.find('div', {'id' : 'pagetitle'})
+        a = pt.find('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"$"))
+        self.story.setMetadata('title',a.string)
+        
+        # Find authorid and URL from... author url.
+        a = pt.find('a', href=re.compile(r"viewuser.php\?uid=\d+"))
+        self.story.setMetadata('authorId',a['href'].split('=')[1])
+        self.story.setMetadata('authorUrl','http://'+self.host+'/'+a['href'])
+        self.story.setMetadata('author',a.string)
+
+        rating=pt.text.split('[')[1].split(']')[0]
+        self.story.setMetadata('rating', rating)
+
+        # Find the chapters:
+        for chapter in soup.findAll('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"&chapter=\d+$")):
+            # just in case there's tags, like <i> in chapter titles.
+            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/'+chapter['href']+addurl))
+
+        self.story.setMetadata('numChapters',len(self.chapterUrls))
+
+        # eFiction sites don't help us out a lot with their meta data
+        # formating, so it's a little ugly.
+
+        # utility method
+        def defaultGetattr(d,k):
+            try:
+                return d[k]
+            except:
+                return ""
+		
+        info = soup.find('div', {'class' : 'small'})
+		
+        word=info.find(text=re.compile("Word count:")).split(':')
+        self.story.setMetadata('numWords', word[1])
+		
+        cats = info.findAll('a',href=re.compile(r'browse.php\?type=categories&id=\d'))
+        for cat in cats:
+            if "General" != cat.string:
+                self.story.addToList('category',cat.string)
+				
+        chars = info.findAll('a',href=re.compile(r'browse.php\?type=characters&charid=\d'))
+        for char in chars:
+            self.story.addToList('characters',char.string)
+			
+        completed=info.find(text=re.compile("Completed: Yes"))
+        if completed != None:
+            self.story.setMetadata('status', 'Completed')
+        else:
+            self.story.setMetadata('status', 'In-Progress')
+			
+        date=soup.find('div',{'class' : 'bottom'})
+        pd=date.find(text=re.compile("Published:")).string.split(': ')
+        self.story.setMetadata('datePublished', makeDate(stripHTML(pd[1].split(' U')[0]), self.dateformat))
+        self.story.setMetadata('dateUpdated', makeDate(stripHTML(pd[2]), self.dateformat))
+		
+        # <span class="label">Rated:</span> NC-17<br /> etc
+        labels = soup.findAll('span',{'class':'label'})
+        pub=0
+        for labelspan in labels:
+            value = labelspan.nextSibling
+            label = labelspan.string
+
+            if 'Genres' in label:
+                genres = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=1'))
+                for genre in genres:
+                    self.story.addToList('genre',genre.string)
+
+            if 'Warnings' in label:
+                warnings = labelspan.parent.findAll('a',href=re.compile(r'browse.php\?type=class&type_id=2'))
+                for warning in warnings:
+                    self.story.addToList('warnings',warning.string)
+
+        try:
+            # Find Series name from series URL.
+            a = soup.find('a', href=re.compile(r"viewseries.php\?seriesid=\d+"))
+            series_name = a.string
+            series_url = 'http://'+self.host+'/'+a['href']
+
+            # use BeautifulSoup HTML parser to make everything easier to find.
+            seriessoup = bs.BeautifulSoup(self._fetchUrl(series_url))
+            storyas = seriessoup.findAll('a', href=re.compile(r'^viewstory.php\?sid=\d+$'))
+            i=1
+            for a in storyas:
+                if a['href'] == ('viewstory.php?sid='+self.story.getMetadata('storyId')):
+                    self.setSeries(series_name, i)
+                    break
+                i+=1
+            
+        except:
+            # I find it hard to care if the series parsing fails
+            pass
+			
+        info.extract()
+        summary = soup.find('div', {'class' : 'content'})
+        self.setDescription(url,summary)
+            
+    # grab the text for an individual chapter.
+    def getChapterText(self, url):
+
+        logging.debug('Getting chapter text from: %s' % url)
+
+        soup = bs.BeautifulStoneSoup(self._fetchUrl(url),
+                                     selfClosingTags=('br','hr')) # otherwise soup eats the br/hr tags.
+        
+        div = soup.find('div', {'id' : 'story'})
+
+        if None == div:
+            raise exceptions.FailedToDownload("Error downloading Chapter: %s!  Missing required element!" % url)
+    
+        return self.utf8FromSoup(url,div)
+
@@ -22,6 +22,10 @@ import logging
 import urllib
 import urllib2 as u2
 import urlparse as up
+from functools import partial
+
+from .. import BeautifulSoup as bs
+from ..htmlcleanup import stripHTML

 try:
    from google.appengine.api import apiproxy_stub_map
@@ -66,8 +70,7 @@ class BaseSiteAdapter(Configurable):
    def __init__(self, config, url):
        self.config = config
        Configurable.__init__(self, config)
-        self.addConfigSection(self.getSiteDomain())
-        self.addConfigSection("overrides")
+        self.setSectionOrder(self.getConfigSection())
        
        self.username = "NoneGiven" # if left empty, site doesn't return any message at all.
        self.password = ""
@@ -77,11 +80,13 @@ class BaseSiteAdapter(Configurable):
        self.storyDone = False
        self.metadataDone = False
        self.story = Story()
-        self.story.setMetadata('site',self.getSiteDomain())
+        self.story.setMetadata('site',self.getConfigSection())
        self.story.setMetadata('dateCreated',datetime.datetime.now())
        self.chapterUrls = [] # tuples of (chapter title,chapter url)
        self.chapterFirst = None
        self.chapterLast = None
+        self.oldchapters = None
+        self.oldimgs = None
        ## order of preference for decoding.
        self.decode = ["utf8",
                       "Windows-1252"] # 1252 is a superset of
@@ -150,6 +155,12 @@ class BaseSiteAdapter(Configurable):
                         headers=headers)
        return self._decode(self.opener.open(req).read())

+    def _fetchUrlRaw(self, url, parameters=None):
+        if parameters != None:
+            return self.opener.open(url,urllib.urlencode(parameters)).read()
+        else:
+            return self.opener.open(url).read()
+    
    # parameters is a dict()
    def _fetchUrl(self, url, parameters=None):
        if self.getConfig('slow_down_sleep_time'):
@@ -159,10 +170,7 @@ class BaseSiteAdapter(Configurable):
        for sleeptime in [0, 0.5, 4, 9]:
            time.sleep(sleeptime)	
            try:
-                if parameters:
-                    return self._decode(self.opener.open(url,urllib.urlencode(parameters)).read())
-                else:
-                    return self._decode(self.opener.open(url).read())
+                return self._decode(self._fetchUrlRaw(url,parameters))
            except Exception, e:
                excpt=e
                logging.warn("Caught an exception reading URL: %s  Exception %s."%(unicode(url),unicode(e)))
@@ -182,15 +190,34 @@ class BaseSiteAdapter(Configurable):
    def getStory(self):
        if not self.storyDone:
            self.getStoryMetadataOnly()
+
            for index, (title,url) in enumerate(self.chapterUrls):
                if (self.chapterFirst!=None and index < self.chapterFirst) or \
                        (self.chapterLast!=None and index > self.chapterLast):
                    self.story.addChapter(removeEntities(title),
                                          None)
                else:
+                    if self.oldchapters and index < len(self.oldchapters):
+                        data = self.utf8FromSoup(None,
+                                                 self.oldchapters[index],
+                                                 partial(cachedfetch,self._fetchUrlRaw,self.oldimgs))
+                    else:
+                        data = self.getChapterText(url)
                    self.story.addChapter(removeEntities(title),
-                                          removeEntities(self.getChapterText(url)))
+                                          removeEntities(data))
            self.storyDone = True
+            
+            # include image, but no cover from story, add default_cover_image cover.
+            if self.getConfig('include_images') and \
+                    not self.story.cover and \
+                    self.getConfig('default_cover_image'):
+                self.story.addImgUrl(self,
+                                     None,
+                                     #self.getConfig('default_cover_image'),
+                                     self.story.formatFileName(self.getConfig('default_cover_image'),
+                                                               self.getConfig('allow_unsafe_filename')),
+                                     self._fetchUrlRaw,
+                                     cover=True)
        return self.story

    def getStoryMetadataOnly(self):
@@ -206,6 +233,11 @@ class BaseSiteAdapter(Configurable):
        "Needs to be overriden in each adapter class."
        return 'no such domain'
    
+    @classmethod
+    def getConfigSection(cls):
+        "Only needs to be overriden if != site domain."
+        return cls.getSiteDomain()
+    
    ## URL pattern validation is done *after* picking an adaptor based
    ## on domain instead of *as* the adaptor selector so we can offer
    ## the user example(s) for that particular site.
@@ -235,6 +267,80 @@ class BaseSiteAdapter(Configurable):
        if self.getConfig('collect_series'):
            self.story.setMetadata('series','%s [%s]'%(name, num))

+    def setDescription(self,url,svalue):
+        #print("\n\nsvalue:\n%s\n"%svalue)
+        if self.getConfig('keep_summary_html'):
+            if isinstance(svalue,str) or isinstance(svalue,unicode):
+                svalue = bs.BeautifulSoup(svalue)
+            self.story.setMetadata('description',self.utf8FromSoup(url,svalue))
+        else:
+            self.story.setMetadata('description',stripHTML(svalue))
+        #print("\n\ndescription:\n"+self.story.getMetadata('description')+"\n\n")
+
+    # This gives us a unicode object, not just a string containing bytes.
+    # (I gave soup a unicode string, you'd think it could give it back...)
+    # Now also does a bunch of other common processing for us.
+    def utf8FromSoup(self,url,soup,fetch=None):
+        if not fetch:
+            fetch=self._fetchUrlRaw
+
+        acceptable_attributes = ['href','name']
+        #print("include_images:"+self.getConfig('include_images'))
+        if self.getConfig('include_images'):
+            acceptable_attributes.extend(('src','alt','longdesc'))
+            for img in soup.findAll('img'):
+                # some pre-existing epubs have img tags that had src stripped off.
+                if img.has_key('src'):
+                    img['longdesc']=img['src']
+                    img['src']=self.story.addImgUrl(self,url,img['src'],fetch,
+                                                    coverexclusion=self.getConfig('cover_exclusion_regexp'))
+
+        for attr in soup._getAttrMap().keys():
+            if attr not in acceptable_attributes:
+                del soup[attr] ## strip all tag attributes except href and name
+                
+        for t in soup.findAll(recursive=True):
+            for attr in t._getAttrMap().keys():
+                if attr not in acceptable_attributes:
+                    del t[attr] ## strip all tag attributes except href and name
+
+            # these are not acceptable strict XHTML.  But we do already have 
+	    # CSS classes of the same names defined
+            if t.name in ('u'):
+                t['class']=t.name
+                t.name='span'
+            if t.name in ('center'):
+                t['class']=t.name
+                t.name='div'
+	    # removes paired, but empty tags.
+            if t.string != None and len(t.string.strip()) == 0 :
+                t.extract()
+                
+        retval = soup.__str__('utf8').decode('utf-8')
+
+        if self.getConfig('replace_hr'):
+            # replacing a self-closing tag with a container tag in the
+            # soup is more difficult than it first appears.  So cheat.
+            retval = retval.replace("<hr />","<div class='center'>* * *</div>")
+
+        if self.getConfig('nook_img_fix'):
+            # if the <img> tag doesn't have a div or a p around it,
+            # nook gets confused and displays it on every page after
+            # that under the text for the rest of the chapter.
+            retval = re.sub(r"(?!<(div|p)>)\s*(?P<imgtag><img[^>]+>)\s*(?!</(div|p)>)",
+                            "<div>\g<imgtag></div>",retval)
+            
+        # Don't want body tags in chapter html--writers add them.
+        # This is primarily for epub updates.
+        return re.sub(r"</?body>\r?\n?","",retval)
+
+def cachedfetch(realfetch,cache,url):
+    if url in cache:
+        print("cache hit")
+        return cache[url]
+    else:
+        return realfetch(url)
+    
 fullmon = {"January":"01", "February":"02", "March":"03", "April":"04", "May":"05",
           "June":"06","July":"07", "August":"08", "September":"09", "October":"10",
           "November":"11", "December":"12" }
@@ -245,7 +351,9 @@ def makeDate(string,format):

    # fudge english month names for people who's locale is set to
    # non-english.  All our current sites date in english, even if
-    # there's non-english content.
+    # there's non-english content. -- ficbook.net now makes that a
+    # lie.  It has to do something even more complicated to get
+    # Russian month names correct everywhere.
    do_abbrev = "%b" in format
        
    if "%B" in format or do_abbrev:
@@ -259,24 +367,3 @@ def makeDate(string,format):
            
    return datetime.datetime.strptime(string,format)

-acceptable_attributes = ['href','name']
-
-# this gives us a unicode object, not just a string containing bytes.
-# (I gave soup a unicode string, you'd think it could give it back...)
-def utf8FromSoup(soup):
-    for t in soup.findAll(recursive=True):
-        for attr in t._getAttrMap().keys():
-            if attr not in acceptable_attributes:
-                del t[attr] ## strip all tag attributes except href and name
-        # these are not acceptable strict XHTML.  But we do already have 
-	# CSS classes of the same names defined in constants.py
-	if t.name in ('u'):
-            t['class']=t.name
-            t.name='span'
-        if t.name in ('center'):
-            t['class']=t.name
-            t.name='div'
-	# removes paired, but empty tags.
-        if t.string != None and len(t.string.strip()) == 0 :
-            t.extract()
-    return soup.__str__('utf8').decode('utf-8')
@@ -21,16 +21,21 @@ import ConfigParser
 # inherit from Configurable.  The config file(s) uses ini format:
 # [sections] with key:value settings.
 #
-# There's a [defaults] section which is overriden by the writer's
-# section [epub], which is overriden by the adapter's section for each
-# site.
+# writer does [defaults], [www.whofic.com], [epub], [www.whofic.com:epub], [overrides]
+#
+# Until a write is created, the adapter only has [defaults], [www.whofic.com], [overrides]
 #
 # [defaults]
 # titlepage_entries: category,genre, status
-# [epub]
-# titlepage_entries: category,genre, status,datePublished,dateUpdated,dateCreated
 # [www.whofic.com]
 # titlepage_entries: category,genre, status,dateUpdated,rating
+# [epub]
+# titlepage_entries: category,genre, status,datePublished,dateUpdated,dateCreated
+# [www.whofic.com:epub]
+# titlepage_entries: category,genre, status,datePublished
+# [overrides]
+# titlepage_entries: category
+

 class Configurable(object):

@@ -38,6 +43,14 @@ class Configurable(object):
        self.config = config
        self.sectionslist = ['defaults']

+    def setSectionOrder(self,site,fileform=None):
+        self.sectionslist = ['defaults']
+        self.addConfigSection(site)
+        if fileform:
+            self.addConfigSection(fileform)
+            self.addConfigSection(site+":"+fileform)
+        self.addConfigSection("overrides")
+
    def addConfigSection(self,section):
        self.sectionslist.insert(0,section)

@@ -0,0 +1,133 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2012, Jim Miller'
+__docformat__ = 'restructuredtext en'
+
+import re, os, traceback
+from zipfile import ZipFile
+from xml.dom.minidom import parseString
+
+from . import BeautifulSoup as bs
+
+def get_dcsource(inputio):
+    return get_update_data(inputio,getfilecount=False,getsoups=False)[0]
+
+def get_dcsource_chaptercount(inputio):
+    return get_update_data(inputio,getfilecount=True,getsoups=False)[:2] # (source,filecount)
+
+def get_update_data(inputio,
+                    getfilecount=True,
+                    getsoups=True):
+    epub = ZipFile(inputio, 'r')
+
+    ## Find the .opf file.
+    container = epub.read("META-INF/container.xml")
+    containerdom = parseString(container)
+    rootfilenodelist = containerdom.getElementsByTagName("rootfile")
+    rootfilename = rootfilenodelist[0].getAttribute("full-path")
+
+    contentdom = parseString(epub.read(rootfilename))
+    firstmetadom = contentdom.getElementsByTagName("metadata")[0]
+    try:
+        source=firstmetadom.getElementsByTagName("dc:source")[0].firstChild.data.encode("utf-8")
+    except:
+        source=None
+
+    ## Save the path to the .opf file--hrefs inside it are relative to it.
+    relpath = get_path_part(rootfilename)
+            
+    filecount = 0
+    soups = [] # list of xhmtl blocks
+    images = {} # dict() longdesc->data
+    if getfilecount:
+        # spin through the manifest--only place there are item tags.
+        for item in contentdom.getElementsByTagName("item"):
+            # First, count the 'chapter' files.  FFDL uses file0000.xhtml,
+            # but can also update epubs downloaded from Twisting the
+            # Hellmouth, which uses chapter0.html.
+            if( item.getAttribute("media-type") == "application/xhtml+xml" ):
+                href=relpath+item.getAttribute("href")
+                #print("---- item href:%s path part: %s"%(href,get_path_part(href)))
+                if re.match(r'.*/(file|chapter)\d+\.x?html',href):
+                    if getsoups:
+                        soup = bs.BeautifulSoup(epub.read(href).decode("utf-8"))
+                        for img in soup.findAll('img'):
+                            newsrc=''
+                            longdesc=''
+                            try:
+                                newsrc=get_path_part(href)+img['src']
+                                # remove all .. and the path part above it, if present.
+                                # Most for epubs edited by Sigil.
+                                newsrc = re.sub(r"([^/]+/\.\./)","",newsrc)
+                                longdesc=img['longdesc']
+                                data = epub.read(newsrc)
+                                images[longdesc] = data
+                                img['src'] = img['longdesc']
+                            except Exception as e:
+                                print("Image %s not found!\n(originally:%s)"%(newsrc,longdesc))
+                                print("Exception: %s"%(unicode(e)))
+                                traceback.print_exc()
+                        soup = soup.find('body')
+                        # ffdl epubs have chapter title h3
+                        h3 = soup.find('h3')
+                        if h3:
+                            h3.extract()
+                        # TtH epubs have chapter title h2
+                        h2 = soup.find('h2')
+                        if h2:
+                            h2.extract()
+                            
+                        soups.append(soup)
+                        
+                    filecount+=1
+
+    for k in images.keys():
+        print("\tlongdesc:%s\n\tData len:%s\n"%(k,len(images[k])))
+    return (source,filecount,soups,images)
+
+def get_path_part(n):
+    relpath = os.path.dirname(n)
+    if( len(relpath) > 0 ):
+        relpath=relpath+"/"
+    return relpath
+
+def get_story_url_from_html(inputio,_is_good_url=None):
+
+    #print("get_story_url_from_html called")
+    epub = ZipFile(inputio, 'r')
+
+    ## Find the .opf file.
+    container = epub.read("META-INF/container.xml")
+    containerdom = parseString(container)
+    rootfilenodelist = containerdom.getElementsByTagName("rootfile")
+    rootfilename = rootfilenodelist[0].getAttribute("full-path")
+
+    contentdom = parseString(epub.read(rootfilename))
+    #firstmetadom = contentdom.getElementsByTagName("metadata")[0]
+
+    ## Save the path to the .opf file--hrefs inside it are relative to it.
+    relpath = get_path_part(rootfilename)
+            
+    # spin through the manifest--only place there are item tags.
+    for item in contentdom.getElementsByTagName("item"):
+        # First, count the 'chapter' files.  FFDL uses file0000.xhtml,
+        # but can also update epubs downloaded from Twisting the
+        # Hellmouth, which uses chapter0.html.
+        #print("---- item:%s"%item)
+        if( item.getAttribute("media-type") == "application/xhtml+xml" ):
+            filehref=relpath+item.getAttribute("href")
+            soup = bs.BeautifulSoup(epub.read(filehref).decode("utf-8"))
+            for link in soup.findAll('a',href=re.compile(r'^http.*')):
+                ahref=link['href']
+                #print("href:(%s)"%ahref)
+                # hack for bad ficsaver ffnet URLs.
+                m = re.match(r"^http://www.fanfiction.net/s(?P<id>\d+)//$",ahref)
+                if m != None:
+                    ahref="http://www.fanfiction.net/s/%s/1/"%m.group('id')
+                if _is_good_url == None or _is_good_url(ahref):
+                    return ahref
+    return None
@@ -34,12 +34,16 @@ class InvalidStoryURL(Exception):
        return "Bad Story URL: (%s) for site: (%s) Example: (%s)" % (self.url, self.domain, self.example)

 class FailedToLogin(Exception):
-    def __init__(self,url,username):
+    def __init__(self,url, username, passwdonly=False):
        self.url=url
        self.username=username
+        self.passwdonly=passwdonly
        
    def __str__(self):
-        return "Failed to Login for URL: (%s) with username: (%s)" % (self.url, self.username)
+        if self.passwdonly:
+            return "URL Failed, password required: (%s) " % (self.url)
+        else:
+            return "Failed to Login for URL: (%s) with username: (%s)" % (self.url, self.username)

 class AdultCheckRequired(Exception):
    def __init__(self,url):
@@ -19,15 +19,31 @@ import re

 def _unirepl(match):
    "Return the unicode string for a decimal number"
-    if match.group(1)=='x':
+    if match.group(1).startswith('x'):
        radix=16
+        s = match.group(1)[1:]
    else:
        radix=10
-    value = int(match.group(2), radix )
-    return unichr(value)
+        s = match.group(1)
+    try:
+        value = int(s, radix)
+        retval = "%s%s"%(unichr(value),match.group(2))
+    except:
+        # This way, at least if there's more of entities out there
+        # that fail, it doesn't blow the entire download.
+        print "Numeric entity translation failed, skipping: &#x%s%s"%(match.group(1),match.group(2))
+        retval = ""
+    return retval

 def _replaceNumberEntities(data):
-    p = re.compile(r'&#(x?)([0-9a-fA-F]+);')
+    # The same brokenish entity parsing in SGMLParser that inserts ';'
+    # after non-entities will also insert ';' incorrectly after number
+    # entities, including part of the next word if it's a-z.
+    # "Don't&#8212ever&#8212do&#8212that&#8212again," becomes
+    # "Don't&#8212e;ver&#8212d;o&#8212;that&#8212a;gain,"
+    # Also need to allow for 5 digit decimal entities &#27861;
+    # Last expression didn't allow for 2 digit hex correctly: &#xE9;
+    p = re.compile(r'&#(x[0-9a-fA-F]{,4}|[0-9]{,5})([0-9a-fA-F]*?);')
    return p.sub(_unirepl, data)

 def _replaceNotEntities(data):
@@ -50,9 +66,6 @@ def removeAllEntities(text):
    return removeEntities(text).replace('&lt;', '<').replace('&gt;', '>').replace('&amp;', '&')

 def removeEntities(text):
-
-    # replace numeric versions of [&<>] with named versions,
-    # then replace named versions with actual characters,
    
    if text is None:
        return ""
@@ -67,6 +80,8 @@ def removeEntities(text):
        except UnicodeEncodeError, e:
            t = text
    text = t 
+    # replace numeric versions of [&<>] with named versions,
+    # then replace named versions with actual characters,
    text = re.sub(r'&#0*38;','&amp;',text)
    text = re.sub(r'&#0*60;','&lt;',text)
    text = re.sub(r'&#0*62;','&gt;',text)
@@ -16,9 +16,131 @@
 #

 import os, re
+import urlparse
+import string
+from math import floor

 from htmlcleanup import conditionalRemoveEntities, removeAllEntities

+# Create convert_image method depending on which graphics lib we can
+# load.  Preferred: calibre, PIL, none
+try:
+    from calibre.utils.magick import Image
+
+    def convert_image(url,data,sizes,grayscale):
+        export = False
+        img = Image()
+        img.load(data)
+        
+        owidth, oheight = img.size
+        nwidth, nheight = sizes
+        scaled, nwidth, nheight = fit_image(owidth, oheight, nwidth, nheight)
+        if scaled:
+            img.size = (nwidth, nheight)
+            export = True
+            
+        if grayscale and img.type != "GrayscaleType":
+            img.type = "GrayscaleType"
+            export = True
+
+        if normalize_format_name(img.format) != "jpg":
+            export = True
+
+        if export:
+            return (img.export('JPG'),'jpg','image/jpeg')
+        else:
+            print("image used unchanged")
+            return (data,'jpg','image/jpeg')
+        
+except:
+
+    # No calibre routines, try for PIL for CLI.
+    try:
+        import Image
+        from StringIO import StringIO
+        def convert_image(url,data,sizes,grayscale):
+            
+            export = False
+            img = Image.open(StringIO(data))
+            
+            owidth, oheight = img.size
+            nwidth, nheight = sizes
+            scaled, nwidth, nheight = fit_image(owidth, oheight, nwidth, nheight)
+            if scaled:
+                img = img.resize((nwidth, nheight),Image.ANTIALIAS)
+                export = True
+                
+            if grayscale and img.mode != "L":
+                img = img.convert("L")
+                export = True
+
+            if normalize_format_name(img.format) != "jpg":
+                if img.mode == "P":
+                    # convert pallete gifs to RGB so jpg save doesn't fail.
+                    img = img.convert("RGB")
+                export = True
+
+            if export:
+                outsio = StringIO()
+                img.save(outsio,'JPEG')
+                return (outsio.getvalue(),'jpg','image/jpeg')
+            else:
+                print("image used unchanged")
+                return (data,'jpg','image/jpeg')
+        
+    except:
+
+        # No calibre or PIL, simple pass through with mimetype.
+        imagetypes = {
+            'jpg':'image/jpeg',
+            'jpeg':'image/jpeg',
+            'png':'image/png',
+            'gif':'image/gif',
+            'svg':'image/svg+xml',
+            }
+
+        def convert_image(url,data,sizes,grayscale):
+            ext=url[url.rfind('.')+1:].lower()
+            return (data,ext,imagetypes[ext])
+        
+def normalize_format_name(fmt):
+    if fmt:
+        fmt = fmt.lower()
+        if fmt == 'jpeg':
+            fmt = 'jpg'
+    return fmt
+
+def fit_image(width, height, pwidth, pheight):
+    '''
+    Fit image in box of width pwidth and height pheight.
+    @param width: Width of image
+    @param height: Height of image
+    @param pwidth: Width of box
+    @param pheight: Height of box
+    @return: scaled, new_width, new_height. scaled is True iff new_width and/or new_height is different from width or height.
+    '''
+    scaled = height > pheight or width > pwidth
+    if height > pheight:
+        corrf = pheight/float(height)
+        width, height = floor(corrf*width), pheight
+    if width > pwidth:
+        corrf = pwidth/float(width)
+        width, height = pwidth, floor(corrf*height)
+    if height > pheight:
+        corrf = pheight/float(height)
+        width, height = floor(corrf*width), pheight
+
+    return scaled, int(width), int(height)
+
+try:
+    # doesn't really matter what, just checking for appengine.
+    from google.appengine.api import apiproxy_stub_map
+
+    is_appengine = True
+except:
+    is_appengine = False
+
+
 # The list comes from ffnet, the only multi-language site we support
 # at the time of writing.  Values are taken largely from pycountry,
 # but with some corrections and guesses.
@@ -69,10 +191,13 @@ class Story:
        try:
            self.metadata = {'version':os.environ['CURRENT_VERSION_ID']}
        except:
-            self.metadata = {'version':'4.3'}
+            self.metadata = {'version':'4.4'}
        self.replacements = []
        self.chapters = [] # chapters will be tuples of (title,html)
+        self.imgurls = []
+        self.imgtuples = []
        self.listables = {} # some items (extratags, category, warnings & genres) are also kept as lists.
+        self.cover=None

    def setMetadata(self, key, value):
        ## still keeps &lt; &lt; and &amp;
@@ -91,9 +216,9 @@ class Story:
        for (p,v) in self.replacements:
            if (isinstance(value,str) or isinstance(value,unicode)) and re.match(p,value):
                value = re.sub(p,v,value)                
-        return value;
+        return value
        
-    def getMetadata(self, key, removeallentities=False):
+    def getMetadata(self, key, removeallentities=False, doreplacements=True):
        value = None
        if self.getLists().has_key(key):
            value = ', '.join(self.getList(key))
@@ -107,21 +232,22 @@ class Story:
                if key == "datePublished" or key == "dateUpdated":
                 value = value.strftime("%Y-%m-%d")

-        value=self.doReplacments(value)
+        if doreplacements:
+            value=self.doReplacments(value)
        if removeallentities and value != None:
            return removeAllEntities(value)
        else:
            return value

-    def getAllMetadata(self, removeallentities=False):
+    def getAllMetadata(self, removeallentities=False, doreplacements=True):
        '''
        All single value *and* list value metadata as strings.
        '''
        allmetadata = {}
        for k in self.metadata.keys():
-            allmetadata[k] = self.getMetadata(k, removeallentities)
+            allmetadata[k] = self.getMetadata(k, removeallentities, doreplacements)
        for l in self.listables.keys():
-            allmetadata[l] = self.getMetadata(l, removeallentities)
+            allmetadata[l] = self.getMetadata(l, removeallentities, doreplacements)

        return allmetadata
        
@@ -153,6 +279,115 @@ class Story:
    def getChapters(self):
        "Chapters will be tuples of (title,html)"
        return self.chapters
+
+    def formatFileName(self,template,allowunsafefilename=True):
+        values = origvalues = self.getAllMetadata()
+        # fall back default:
+        if not template:
+            template="${title}-${siteabbrev}_${storyId}${formatext}"
+
+        if not allowunsafefilename:
+            values={}
+            pattern = re.compile(r"[^a-zA-Z0-9_\. \[\]\(\)&'-]+")
+            for k in origvalues.keys():
+                values[k]=re.sub(pattern,'_', removeAllEntities(self.getMetadata(k)))
+
+        return string.Template(template).substitute(values).encode('utf8')
+
+    # pass fetch in from adapter in case we need the cookies collected
+    # as well as it's a base_story class method.
+    def addImgUrl(self,configurable,parenturl,url,fetch,cover=False,coverexclusion=None):
+        
+        url = url.strip() # ran across an image with a space in the
+                          # src. Browser handled it, so we'd better, too.
+
+        # appengine (web version) isn't allowed to do images--just
+        # gets too big too fast and breaks things.
+        if is_appengine:
+            return
+        
+        if url.startswith("http") or url.startswith("file") or parenturl == None:
+            imgurl = url
+        else:
+            parsedUrl = urlparse.urlparse(parenturl)
+            if url.startswith("/") :
+                imgurl = urlparse.urlunparse(
+                    (parsedUrl.scheme,
+                     parsedUrl.netloc,
+                     url,
+                     '','',''))
+            else:
+                toppath=""
+                if parsedUrl.path.endswith("/"):
+                    toppath = parsedUrl.path
+                else:
+                    toppath = parsedUrl.path[:parsedUrl.path.rindex('/')]
+                imgurl = urlparse.urlunparse(
+                    (parsedUrl.scheme,
+                     parsedUrl.netloc,
+                     toppath + '/' + url,
+                     '','',''))
+                #print("\n===========\nparsedUrl.path:%s\ntoppath:%s\nimgurl:%s\n\n"%(parsedUrl.path,toppath,imgurl))
+
+        prefix='ffdl'
+        if imgurl not in self.imgurls:
+            parsedUrl = urlparse.urlparse(imgurl)
+            sizes = [ int(x) for x in configurable.getConfigList('image_max_size') ]
+            try:
+                (data,ext,mime) = convert_image(imgurl,
+                                                fetch(imgurl),
+                                                sizes,
+                                                configurable.getConfig('grayscale_images'))
+            except Exception, e:
+                print("Failed to load or convert image, skipping:\n%s\nException: %s"%(imgurl,e))
+                return "failedtoload"
+            
+            # explicit cover, make the first image.
+            if cover and not configurable.getConfig('never_make_cover'):
+                if len(self.imgtuples) > 0 and 'cover' in self.imgtuples[0]['newsrc']:
+                    # remove existing cover, if there is one.
+                    del self.imgurls[0]
+                    del self.imgtuples[0]
+                self.imgurls.insert(0,imgurl)
+                newsrc = "images/cover.%s"%ext
+                self.cover=newsrc
+                self.imgtuples.insert(0,{'newsrc':newsrc,'mime':mime,'data':data})
+            else:
+                self.imgurls.append(imgurl)
+                # First image, copy not link because calibre will replace with it's cover.
+                # Only if: No cover already AND
+                #          make_firstimage_cover AND
+                #          NOT never_make_cover AND
+                #          either no coverexclusion OR coverexclusion doesn't match
+                if self.cover == None and \
+                        configurable.getConfig('make_firstimage_cover') and \
+                        not configurable.getConfig('never_make_cover') and \
+                        (not coverexclusion or not re.search(coverexclusion,imgurl)):
+                    newsrc = "images/cover.%s"%ext
+                    self.cover=newsrc
+                    self.imgtuples.append({'newsrc':newsrc,'mime':mime,'data':data})
+                    self.imgurls.append(imgurl)
+            
+                newsrc = "images/%s-%s.%s"%(
+                    prefix,
+                    self.imgurls.index(imgurl),
+                    ext)
+                self.imgtuples.append({'newsrc':newsrc,'mime':mime,'data':data})
+                
+            print("\nimgurl:%s\nnewsrc:%s\nimage size:%d\n"%(imgurl,newsrc,len(data)))
+        else:
+            newsrc = self.imgtuples[self.imgurls.index(imgurl)]['newsrc']
+            
+        #print("===============\n%s\nimg url:%s\n============"%(newsrc,self.imgurls[-1]))
+        
+        return newsrc
+
+    def getImgUrls(self):
+        retlist = []
+        for i, url in enumerate(self.imgurls):
+            #parsedUrl = urlparse.urlparse(url)
+            retlist.append(self.imgtuples[i])
+        return retlist
    
    def __str__(self):
        return "Metadata: " +str(self.metadata) + "\nListables: " +str(self.listables) #+ "\nChapters: "+str(self.chapters)
@@ -18,7 +18,6 @@
 import re
 import os.path
 import datetime
-import string
 import StringIO
 import zipfile
 from zipfile import ZipFile, ZIP_DEFLATED
@@ -39,10 +38,7 @@ class BaseStoryWriter(Configurable):

    def __init__(self, config, adapter):
        Configurable.__init__(self, config)
-        self.addConfigSection(adapter.getSiteDomain())
-        self.addConfigSection(self.getFormatName())
-        self.addConfigSection(adapter.getSiteDomain()+":"+self.getFormatName())
-        self.addConfigSection("overrides")
+        self.setSectionOrder(adapter.getConfigSection(),self.getFormatName())
        
        self.adapter = adapter
        self.story = adapter.getStoryMetadataOnly() # only cache the metadata initially.
@@ -122,29 +118,15 @@ class BaseStoryWriter(Configurable):
            return self.getBaseFileName()

    def getBaseFileName(self):
-        return self.formatFileName(self.getConfig('output_filename'))
+        return self.story.formatFileName(self.getConfig('output_filename'),self.getConfig('allow_unsafe_filename'))
    
    def getZipFileName(self):
-        return self.formatFileName(self.getConfig('zip_filename'))
-
-    def formatFileName(self,template):
-        values = origvalues = self.story.getAllMetadata()
-        # fall back default:
-        if not template:
-            template="${title}-${siteabbrev}_${storyId}${formatext}"
-
-        if not self.getConfig('allow_unsafe_filename'):
-            values={}
-            pattern = re.compile(r"[^a-zA-Z0-9_\. \[\]\(\)&'-]+")
-            for k in origvalues.keys():
-                values[k]=re.sub(pattern,'_', removeAllEntities(self.story.getMetadata(k)))
-
-        return string.Template(template).substitute(values).encode('utf8')
+        return self.story.formatFileName(self.getConfig('zip_filename'),self.getConfig('allow_unsafe_filename'))

    def _write(self, out, text):
        out.write(text.encode('utf8'))

-    def writeTitlePage(self, out, START, ENTRY, END, WIDE_ENTRY=None):
+    def writeTitlePage(self, out, START, ENTRY, END, WIDE_ENTRY=None, NO_TITLE_ENTRY=None):
        """
        Write the title page, but only include entries that there's
        metadata for.  START, ENTRY and END are expected to already by
@@ -171,6 +153,12 @@ class BaseStoryWriter(Configurable):
                            label=self.getConfig(entry+"_label")
                        else:
                            label=self.titleLabels[entry]
+
+                        # If the label for the title entry is empty, use the
+                        # 'no title' option if there is one.  
+                        if label == "" and NO_TITLE_ENTRY:
+                           TEMPLATE= NO_TITLE_ENTRY
+                           
                        self._write(out,TEMPLATE.substitute({'label':label,
                                                             'value':self.story.getMetadata(entry)}))

@@ -20,6 +20,7 @@ import string
 import StringIO
 import zipfile
 from zipfile import ZipFile, ZIP_STORED, ZIP_DEFLATED
+import urllib

 ## XML isn't as forgiving as HTML, so rather than generate as strings,
 ## use DOM to generate the XML files.
@@ -57,6 +58,10 @@ class EpubWriter(BaseStoryWriter):

        self.EPUB_TITLE_ENTRY = string.Template('''
 <b>${label}:</b> ${value}<br />
+''')
+
+        self.EPUB_NO_TITLE_ENTRY = string.Template('''
+${value}<br />
 ''')

        self.EPUB_TITLE_PAGE_END = string.Template('''
@@ -84,6 +89,10 @@ class EpubWriter(BaseStoryWriter):

        self.EPUB_TABLE_TITLE_WIDE_ENTRY = string.Template('''
 <tr><td colspan="2"><b>${label}:</b> ${value}</td></tr>
+''')
+
+        self.EPUB_TABLE_NO_TITLE_ENTRY = string.Template('''
+<tr><td colspan="2">${label}${value}</td></tr>
 ''')

        self.EPUB_TABLE_TITLE_PAGE_END = string.Template('''
@@ -252,7 +261,54 @@ class EpubWriter(BaseStoryWriter):
        itemrefs = [] # list of strings -- idrefs from .opfs' spines
        items.append(("ncx","toc.ncx","application/x-dtbncx+xml",None)) ## we'll generate the toc.ncx file,
                                                                   ## but it needs to be in the items manifest.
+
+        if self.getConfig('include_images'):
+            imgcount=0
+            for imgmap in self.story.getImgUrls():
+                imgfile = "OEBPS/"+imgmap['newsrc']
+                outputepub.writestr(imgfile,imgmap['data'])
+                items.append(("image%04d"%imgcount,
+                              imgfile,
+                              imgmap['mime'],
+                              None))
+                imgcount+=1
+
+        
        items.append(("style","OEBPS/stylesheet.css","text/css",None))
+
+        guide = None
+        coverIO = None
+
+        if self.story.cover:
+            # Note that the id of the cover xhmtl *must* be 'cover'
+            # for it to work on Nook.
+            items.append(("cover","OEBPS/cover.xhtml","application/xhtml+xml",None))
+            itemrefs.append("cover")
+            # 
+            # <meta name="cover" content="cover.jpg"/>
+            metadata.appendChild(newTag(contentdom,"meta",{"content":"image0000",
+                                                           "name":"cover"}))
+            # cover stuff for later:
+            # at end of <package>:
+            # <guide>
+            # <reference type="cover" title="Cover" href="Text/cover.xhtml"/>
+            # </guide>
+            guide = newTag(contentdom,"guide")
+            guide.appendChild(newTag(contentdom,"reference",attrs={"type":"cover",
+                                                       "title":"Cover",
+                                                       "href":"OEBPS/cover.xhtml"}))
+            
+            coverIO = StringIO.StringIO()
+            coverIO.write('''
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"><head><title>Cover</title><style type="text/css" title="override_css">
+@page {padding: 0pt; margin:0pt}
+body { text-align: center; padding:0pt; margin: 0pt; }
+div { margin: 0pt; padding: 0pt; }
+</style></head><body><div>
+<img src="%s" alt="cover"/>
+</div></body></html>
+'''%self.story.cover)
+            
        if self.getConfig("include_titlepage"):
            items.append(("title_page","OEBPS/title_page.xhtml","application/xhtml+xml","Title Page"))
            itemrefs.append("title_page")
@@ -283,8 +339,18 @@ class EpubWriter(BaseStoryWriter):
            spine.appendChild(newTag(contentdom,"itemref",
                                     attrs={"idref":itemref,
                                            "linear":"yes"}))
+        # guide only exists if there's a cover.
+        if guide:
+            package.appendChild(guide)
+            
        # write content.opf to zip.
-        outputepub.writestr("content.opf",contentdom.toxml(encoding='utf-8'))
+        contentxml = contentdom.toxml(encoding='utf-8')
+        
+        # tweak for brain damaged Nook STR.  Nook insists on name before content.
+        contentxml = contentxml.replace('<meta content="image0000" name="cover"/>',
+                                        '<meta name="cover" content="image0000"/>')
+        outputepub.writestr("content.opf",contentxml)
+
        contentdom.unlink()
        del contentdom

@@ -320,7 +386,7 @@ class EpubWriter(BaseStoryWriter):
        index=0
        for item in items:
            (id,href,type,title)=item
-            # only items to be skipped, toc.ncx, stylesheet.css, should have no title.
+            # only items to be skipped, cover.xhtml, images, toc.ncx, stylesheet.css, should have no title.
            if title :
                navPoint = newTag(tocncxdom,"navPoint",
                                  attrs={'id':id,
@@ -333,7 +399,7 @@ class EpubWriter(BaseStoryWriter):
                navPoint.appendChild(newTag(tocncxdom,"content",attrs={"src":href}))
                index=index+1
        
-        # write toc.ncs to zip file
+        # write toc.ncx to zip file
        outputepub.writestr("toc.ncx",tocncxdom.toxml(encoding='utf-8'))
        tocncxdom.unlink()
        del tocncxdom
@@ -346,19 +412,26 @@ class EpubWriter(BaseStoryWriter):
            TITLE_PAGE_START  = self.EPUB_TABLE_TITLE_PAGE_START
            TITLE_ENTRY       = self.EPUB_TABLE_TITLE_ENTRY
            WIDE_TITLE_ENTRY  = self.EPUB_TABLE_TITLE_WIDE_ENTRY
+            NO_TITLE_ENTRY    = self.EPUB_TABLE_NO_TITLE_ENTRY
            TITLE_PAGE_END    = self.EPUB_TABLE_TITLE_PAGE_END
        else:
            TITLE_PAGE_START  = self.EPUB_TITLE_PAGE_START
            TITLE_ENTRY       = self.EPUB_TITLE_ENTRY
            WIDE_TITLE_ENTRY  = self.EPUB_TITLE_ENTRY # same, only wide in tables.
+            NO_TITLE_ENTRY    = self.EPUB_NO_TITLE_ENTRY
            TITLE_PAGE_END    = self.EPUB_TITLE_PAGE_END
-        
+
+        if coverIO:
+            outputepub.writestr("OEBPS/cover.xhtml",coverIO.getvalue())
+            coverIO.close()
+            
        titlepageIO = StringIO.StringIO()
        self.writeTitlePage(out=titlepageIO,
                            START=TITLE_PAGE_START,
                            ENTRY=TITLE_ENTRY,
                            WIDE_ENTRY=WIDE_TITLE_ENTRY,
-                            END=TITLE_PAGE_END)
+                            END=TITLE_PAGE_END,
+                            NO_TITLE_ENTRY=NO_TITLE_ENTRY)
        if titlepageIO.getvalue(): # will be false if no title page.
            outputepub.writestr("OEBPS/title_page.xhtml",titlepageIO.getvalue())
        titlepageIO.close()
@@ -384,7 +457,7 @@ class EpubWriter(BaseStoryWriter):
                fullhtml = fullhtml.replace('</p>','</p>\n').replace('<br />','<br />\n')
                outputepub.writestr("OEBPS/file%04d.xhtml"%(index+1),fullhtml.encode('utf-8'))
                del fullhtml
- 
+
 	# declares all the files created by Windows.  otherwise, when
        # it runs in appengine, windows unzips the files as 000 perms.
        for zf in outputepub.filelist:
@@ -49,6 +49,10 @@ class MobiWriter(BaseStoryWriter):

        self.MOBI_TITLE_ENTRY = string.Template('''
 <b>${label}:</b> ${value}<br />
+''')
+
+        self.MOBI_NO_TITLE_ENTRY = string.Template('''
+${value}<br />
 ''')

        self.MOBI_TITLE_PAGE_END = string.Template('''
@@ -75,6 +79,10 @@ class MobiWriter(BaseStoryWriter):

        self.MOBI_TABLE_TITLE_WIDE_ENTRY = string.Template('''
 <tr><td colspan="2"><b>${label}:</b> ${value}</td></tr>
+''')
+
+        self.MOBI_TABLE_NO_TITLE_WIDE_ENTRY = string.Template('''
+<tr><td colspan="2">${value}</td></tr>
 ''')

        self.MOBI_TABLE_TITLE_PAGE_END = string.Template('''
@@ -129,11 +137,13 @@ class MobiWriter(BaseStoryWriter):
            TITLE_PAGE_START  = self.MOBI_TABLE_TITLE_PAGE_START
            TITLE_ENTRY       = self.MOBI_TABLE_TITLE_ENTRY
            WIDE_TITLE_ENTRY  = self.MOBI_TABLE_TITLE_WIDE_ENTRY
+            NO_TITLE_ENTRY    = self.MOBI_TABLE_NO_TITLE_ENTRY
            TITLE_PAGE_END    = self.MOBI_TABLE_TITLE_PAGE_END
        else:
            TITLE_PAGE_START  = self.MOBI_TITLE_PAGE_START
            TITLE_ENTRY       = self.MOBI_TITLE_ENTRY
            WIDE_TITLE_ENTRY  = self.MOBI_TITLE_ENTRY # same, only wide in tables.
+            NO_TITLE_ENTRY    = self.MOBI_NO_TITLE_ENTRY
            TITLE_PAGE_END    = self.MOBI_TITLE_PAGE_END
        
        titlepageIO = StringIO.StringIO()
@@ -141,7 +151,8 @@ class MobiWriter(BaseStoryWriter):
                            START=TITLE_PAGE_START,
                            ENTRY=TITLE_ENTRY,
                            WIDE_ENTRY=WIDE_TITLE_ENTRY,
-                            END=TITLE_PAGE_END)
+                            END=TITLE_PAGE_END,
+                            NO_TITLE_ENTRY=NO_TITLE_ENTRY)
        if titlepageIO.getvalue(): # will be false if no title page.
            files.append(titlepageIO.getvalue())
        titlepageIO.close()
@@ -13,8 +13,23 @@
 # limitations under the License.
 #

+import pickle, copy
 from google.appengine.ext import db

+class ObjectProperty(db.Property):
+	data_type = db.Blob
+
+	def get_value_for_datastore(self, model_instance):
+		value = self.__get__(model_instance, model_instance.__class__)
+		pickled_val = pickle.dumps(value,protocol=pickle.HIGHEST_PROTOCOL)
+		if value is not None: return db.Blob(pickled_val)
+
+	def make_value_from_datastore(self, value):
+		if value is not None: return pickle.loads(value)
+
+	def default_value(self):
+		return copy.copy(self.default)
+
 class DownloadMeta(db.Model):
 	user = db.UserProperty()
 	url = db.StringProperty()
@@ -37,3 +52,12 @@ class DownloadData(db.Model):
 class UserConfig(db.Model):
 	user = db.UserProperty()
 	config = db.BlobProperty()
+
+class SavedMeta(db.Model):
+	url = db.StringProperty()
+	title = db.StringProperty()
+	author = db.StringProperty()
+	date = db.DateTimeProperty(auto_now_add=True)
+	count = db.IntegerProperty()
+	meta = ObjectProperty()
+	
@@ -54,30 +54,28 @@
 	      much easier. </p>
 	  </div>
 	  <!-- put announcements here, h3 is a good title size. -->
-	  <h3>New Russian Language Site ficbook.net</h3>
+	  <h3>New Sites</h3>
 	  <p>
-	    Thanks to Ida Leter's hard work, we now support <a href="http://ficbook.net">ficbook.net</a>, a Russian language fanfiction site.
-	  </p>
-	  <h3>Support for Language, Custom CSS and Replacement of Metadata</h3>
-	  <p>
-	    There's now a 'Language' metadata field that can be filled, if the site supports different languages.  Currently, it's only used
-	    with fanfiction.net and ficbook.net.
+	    We now support www.wraithbait.com and
+	    www.libraryofmoria.com, thanks to Ida for adding these!
 	  </p>
 	  <p>
-	    The CSS included in the HTML and EPUB output formats is now a customizable parameter.
+	    Support for the Wonderful World of MakeBelieve(WWOMB)
+	    archive at
+	    <a href="http://www.squidge.org/peja/cgi-bin/index.php">http://www.squidge.org/peja/cgi-bin/index.php</a>
+	    has also been added.  This does not support other sections of
+	    www.squidge.org, or the other files under www.squidge.org/peja that
+	    aren't in the eFiction instance.
 	  </p>
 	  <p>
-	    There's now a customizable parameter to include a list of regular expressions to replace metadata as you see fit.
-	  </p>
-	  <p>
-	    Examples of how to use both new features can be found in the
-	    <a href="http://www.mobileread.com/forums/showthread.php?p=1962034#post1962034">plugin forum</a>.
-	  </p>
+	    Questions?  Check out our 
+	    <a href="http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderFAQs">FAQs</a>.
+	    </p>
 	  <p>
 	    If you have any problems with this application, please
 	    report them in
 	    the <a href="http://groups.google.com/group/fanfic-downloader">FanFictionDownLoader Google Group</a>.  The
-	    <a href="http://4-3-1.fanfictiondownloader.appspot.com">Previous Version</a> is also available for you to use if necessary.
+	    <a href="http://4-4-8.fanfictiondownloader.appspot.com">Previous Version</a> is also available for you to use if necessary.
 	  </p>
 	  <div id='error'>
 	    {{ error_message }}
@@ -107,6 +105,9 @@
 	  <p>
 	    Or see your personal list of <a href="/recent">previously downloaded fanfics</a>.
 	  </p>
+	  <p>
+	    See a list of downloaded fanfics by all users by <a href="/allrecent">most popular</a> or <a href="/allrecent?bydate=1">most recent</a>.
+	  </p>
 	</div>	
      </form>
      {% else %}
@@ -141,6 +142,13 @@
 	  </p>
      </div>
      <div id='helpbox'>
+	<h3>Supported sites:</h3>
+	<p>
+	  There's a 
+ 	  <a href="http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderSupportedsites">Supported
+ 	  Sites</a> page in our wiki.  If you have a site you'd like
+ 	  to see supported, please check there first.
+	</p>
 	<dl>
 	  <dt>fictionalley.org</dt>
 	  <dd>
@@ -246,24 +254,80 @@
 	    <br /><a href="http://archiveofourown.org/works/76366">http://archiveofourown.org/works/76366</a>.
 	    <br /><a href="http://archiveofourown.org/works/76366/chapters/101584">http://archiveofourown.org/works/76366/chapters/101584</a>.
 	  </dd>
-	</dl>	
 	  <dt>ficbook.net(Russian)</dt>
 	  <dd>
 	    Use the URL of the story, or one of it's chapters, such as
 	    <br /><a href="http://ficbook.net/readfic/93626">http://ficbook.net/readfic/93626</a>.
 	    <br /><a href="http://ficbook.net/readfic/93626/246417#part_content">http://ficbook.net/readfic/93626/246417#part_content</a>.
 	  </dd>
+	  <dt>fanfiction.mugglenet.com</dt>
+	  <dd>
+	    Use the URL of the story's chapter list, such as
+	    <br /><a href="http://fanfiction.mugglenet.com/viewstory.php?sid=123">http://fanfiction.mugglenet.com/viewstory.php?sid=123</a>.
+	  </dd>
+	  <dt>www.hpfandom.net</dt>
+	  <dd>
+	    Use the URL of the story's chapter list, such as
+	    <br /><a href="http://www.hpfandom.net/eff/viewstory.php?sid=123">http://www.hpfandom.net/eff/viewstory.php?sid=123</a>.
+	  </dd>
+	  <dt>thequidditchpitch.org</dt>
+	  <dd>
+	    Use the URL of the story's chapter list, such as
+	    <br /><a href="http://thequidditchpitch.org/viewstory.php?sid=123">http://thequidditchpitch.org/viewstory.php?sid=123</a>.
+	  </dd>
+	  <dt>fanfiction.portkey.org</dt>
+	  <dd>
+	    Use the URL of the story's chapter list, such as
+	    <br /><a href="http://fanfiction.portkey.org/story/123">http://fanfiction.portkey.org/story/123</a>.
+	  </dd>
+	  <dt>nfacommunity.com</dt>
+	  <dd>
+	    Use the URL of the story's chapter list, such as
+	    <br /><a href="http://nfacommunity.com/viewstory.php?sid=1654">http://nfacommunity.com/viewstory.php?sid=1654</a>.
+	  </dd>
+	  <dt>www.midnightwhispers.ca</dt>
+	  <dd>
+	    Use the URL of the story's chapter list, such as
+	    <br /><a href="http://www.midnightwhispers.ca/viewstory.php?sid=1124">http://www.midnightwhispers.ca/viewstory.php?sid=1124</a>.
+	  </dd>
+	  <dt>ksarchive.com</dt>
+	  <dd>
+	    Use the URL of the story's chapter list, such as
+	    <br /><a href="http://ksarchive.com/viewstory.php?sid=1124">http://ksarchive.com/viewstory.php?sid=1124</a>.
+	  </dd>
+	  <dt>archive.skyehawke.com</dt>
+	  <dd>
+	    Use the URL of the story's summary, such as
+	    <br /><a href="http://archive.skyehawke.com/story.php?no=17466">http://archive.skyehawke.com/story.php?no=17466</a>.
+	  </dd>
+	  <dt>www.libraryofmoria.com</dt>
+	  <dd>
+	    Use the URL of the story's summary, such as
+	    <br /><a href="http://www.libraryofmoria.com/a/viewstory.php?sid=434">http://www.libraryofmoria.com/a/viewstory.php?sid=434</a>.
+	  </dd>
+	  <dt>www.wraithbait.com</dt>
+	  <dd>
+	    Use the URL of the story's summary, such as
+	    <br /><a href="http://www.wraithbait.com/viewstory.php?sid=14305">http://www.wraithbait.com/viewstory.php?sid=14305</a>.
+	  </dd>
+	  <dt>www.squidge.org/peja (WWOMB)</dt>
+	  <dd>
+	    Use the URL of the story's summary, such as
+	    <br /><a href="http://www.squidge.org/peja/cgi-bin/viewstory.php?sid=1234">http://www.squidge.org/peja/cgi-bin/viewstory.php?sid=1234</a>.<br />
+	    This is <i>only</i> for squidge.org/peja, not other parts of squidge.org.
+	  </dd>
 	</dl>
 	<p>
 	  A few additional things to know, which will make your life substantially easier:
 	</p>
 	<ol>
 	  <li>
-	    First thing to know: I do not use your Google login and password. In fact, all I know about it is your ID &ndash; password
+	    First thing to know: We do not use your Google login and password. In fact, all we know about it is your ID &ndash; password
 	    is being verified by Google and is absolutely, totally unknown to anyone but you.
 	  </li>
 	  <li>
-	    Small <a href="http://www.sigizmund.com/reading-fanfiction-off-line-in-stanza-and-oth">post written by me</a>
+	    
+	    Small <a href="http://www.sigizmund.com/reading-fanfiction-off-line-in-stanza-and-oth">post written by Roman</a>
 	    &mdash; how to read fiction in Stanza or any other ebook reader.
 	  </li>
 	  <li>
@@ -279,7 +343,7 @@
 	  </li>
 	  <li>
 	    If you think that something that should work in fact doesn't, post a message to
-	    our <a href="http://groups.google.com/group/fanfic-downloader">Google Group</a>. I also encourage you to join it so
+	    our <a href="http://groups.google.com/group/fanfic-downloader">Google Group</a>. we also encourage you to join it so
 	    you will find out about latest updates and fixes as soon as possible
 	  </li>
 	</ol>
@@ -1,6 +1,6 @@
 indexes:

-# AUTOGENERATED
+# notAUTOGENERATED

 # This index.yaml is automatically updated whenever the dev_appserver
 # detects that a new type of query is run.  If you want to manage the
@@ -21,13 +21,8 @@ indexes:
  - name: date
    direction: desc

- kind: DownloadedFanfic
+- kind: SavedMeta
  properties:
-  - name: cleared
-  - name: date
-
- kind: DownloadedFanfic
-  properties:
-  - name: user
+  - name: count
  - name: date
    direction: desc
@@ -303,6 +303,39 @@ class RecentFilesServer(webapp2.RequestHandler):
        path = os.path.join(os.path.dirname(__file__), 'recent.html')
        self.response.out.write(template.render(path, template_values))

+class AllRecentFilesServer(webapp2.RequestHandler):
+    def get(self):
+        user = users.get_current_user()
+        if not user:
+            self.redirect(users.create_login_url(self.request.uri))
+            return
+
+        q = SavedMeta.all()
+        if self.request.get('bydate'):
+            q.order('-date')
+        else:
+            q.order('-count')
+            
+        fics = q.fetch(200)
+        logging.info("Recent fetched %d downloads for user %s."%(len(fics),user.nickname()))
+
+        sendslugs = []
+
+        for fic in fics:
+            ficslug = FicSlug(fic)
+            sendslugs.append(ficslug)
+                
+        template_values = dict(fics = sendslugs, nickname = user.nickname())
+        path = os.path.join(os.path.dirname(__file__), 'allrecent.html')
+        self.response.out.write(template.render(path, template_values))
+
+class FicSlug():
+    def __init__(self,savedmeta):
+        self.url = savedmeta.url
+        self.count = savedmeta.count
+        for k, v in savedmeta.meta.iteritems():
+            setattr(self,k,v)
+        
 class FanfictionDownloader(UserConfigServer):
    def get(self):
        self.post()
@@ -339,7 +372,7 @@ class FanfictionDownloader(UserConfigServer):
                self.redirect("/?error=custom&errtext=%s"%urlEscape("There's an error in your User Configuration: "+str(e)))
                return

-            adapter = adapters.getAdapter(config,url)
+            adapter = adapters.getAdapter(config,url,format)
            logging.info('Created an adaper: %s' % adapter)

            if len(login) > 1:
@@ -385,7 +418,7 @@ class FanfictionDownloader(UserConfigServer):
            template_values = dict(nickname = user.nickname(),
                                   url = url,
                                   format = format,
-                                   site = adapter.getSiteDomain(),
+                                   site = adapter.getConfigSection(),
                                   fic = download,
                                   is_login=is_login,
                                   )
@@ -442,7 +475,7 @@ class FanfictionDownloaderTask(UserConfigServer):

        try:
            config = self.getUserConfig(user)
-            adapter = adapters.getAdapter(config,url)
+            adapter = adapters.getAdapter(config,url,format)

            logging.info('Created an adapter: %s' % adapter)

@@ -464,6 +497,8 @@ class FanfictionDownloaderTask(UserConfigServer):
            download.url = adapter.getStory().getMetadata('storyUrl')
            download.put()

+            allmeta = adapter.getStory().getAllMetadata(removeallentities=True,doreplacements=False)
+
            outbuffer = StringIO()
            writer.writeStory(outbuffer)
            data = outbuffer.getvalue()
@@ -495,6 +530,21 @@ class FanfictionDownloaderTask(UserConfigServer):
            download.completed=True
            download.put()

+            smetal = SavedMeta.all().filter('url =', allmeta['storyUrl'] ).fetch(1)
+            if smetal and smetal[0]:
+                smeta = smetal[0]
+                smeta.count += 1
+            else:
+                smeta=SavedMeta()
+                smeta.count = 1
+
+            smeta.url = allmeta['storyUrl']
+            smeta.title = allmeta['title']
+            smeta.author = allmeta['author']
+            smeta.meta = allmeta
+            smeta.date = datetime.datetime.now()
+            smeta.put()
+            
            logging.info("Download finished OK")
            del data

@@ -563,6 +613,7 @@ app = webapp2.WSGIApplication([('/', MainHandler),
                               ('/fdown', FanfictionDownloader),
                               (r'/file.*', FileServer),
                               ('/status', FileStatusServer),
+                               ('/allrecent', AllRecentFilesServer),
                               ('/recent', RecentFilesServer),
                               ('/editconfig', EditConfigServer),
                               ('/clearrecent', ClearRecentServer),
@@ -108,10 +108,6 @@ extratags: FanFiction
 ## useful if pulling large numbers of stories or if the site is slow.
 #slow_down_sleep_time:0.5

-## output background color--only used by html and epub (and ignored in
-## epub by many readers).  Must be hex code, # will be added.
-background_color: ffffff
-
 ## Use regular expressions to find and replace (or remove) metadata.
 ## For example, you could change Sci-Fi=>SF, remove *-Centered tags,
 ## etc.  See http://docs.python.org/library/re.html (look for re.sub)
@@ -126,6 +122,25 @@ background_color: ffffff
 # (.*)Great(.*)=>\1Moderate\2
 # .*-Centered=>

+## Some readers don't show horizontal rule (<hr />) tags correctly.
+## This replaces them all with a centered '* * *'.  (Note centering
+## doesn't work on some devices either.)
+#replace_hr: false
+
+## If you have the Generate Cover plugin installed, you can use the
+## generate_cover_settings parameter to intelligently decide which GC
+## setting to run.  There are three parts 1) a template of which
+## metadata part(s) to look at, 2) a regular expression to match the
+## template, and 3) the name of the GC setting to use, which must
+## match exactly.  Use this parameter in [defaults], or by site eg,
+## [www.ficwad.com]
+## Make sure to keep at least one space at the start of each line and
+## to escape % to %%, if used.
+## template => regexp to match => GC Setting to use.
+#generate_cover_settings:
+# ${category} => Buffy:? [tT]he Vampire Slayer => BuffyCover
+# ${category} => Star Trek => StarTrekCover
+
 ## Each output format has a section that overrides [defaults]
 [html]

@@ -160,13 +175,6 @@ titlepage_entries: series,category,genre,language,status,datePublished,dateUpdat
 ## use \r\n for line endings, the windows convention.  text output only.
 windows_eol: true

-[txt]
-## Add URLs since there aren't links.
-titlepage_entries: series,category,genre,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,numWords,site,storyUrl, authorUrl, description
-
-## use \r\n for line endings, the windows convention.  text output only.
-windows_eol: true
-
 [epub]

 ## epub carries the TOC in metadata.
@@ -213,11 +221,61 @@ output_css:
 .u        {text-decoration: underline;}
 .bold     {font-weight: bold;}

+## include images from img tags in the body and summary of
+## stories.  Images will be converted to jpg for size if possible.
+#include_images:false
+
+## If not set, the summary will have all html stripped for safety.
+## Both this and include_images must be true to get images in the
+## summary.
+#keep_summary_html:false
+
+## If set, the first image found will be made the cover image.  If
+## keep_summary_html is true, any images in summary will be before any
+## in chapters.
+#make_firstimage_cover: false
+
+## If set, the epub will never have a cover, even include_images is on
+## and the site has specific cover images.
+#never_make_cover: false
+
+## If set, and there isn't already a cover image from the adapter or
+## from make_firstimage_cover, this image will be made the cover.
+## It can be either a 'file:' or 'http:' url.
+## Note that if you enable make_firstimage_cover in [epub], but want
+## to use default_cover_image for a specific site, use the site:format
+## section, for example: [www.ficwad.com:epub]
+## default_cover_image is a python string Template string with
+## ${title}, ${author} etc, same as titlepage_entries. Unless
+## allow_unsafe_filename is true, invalid filename chars will be
+## removed from metadata fields
+#default_cover_image:file:///C:/Users/username/Desktop/nook/images/icon.png
+#default_cover_image:file:///C:/Users/username/Desktop/nook/images/${title}/icon.png
+#default_cover_image:http://www.somesite.com/someimage.gif
+
+## some sites include images that we don't ever want becoming the
+## cover image.  This lets you exclude them.
+#cover_exclusion_regexp:/stories/999/images/.*?_trophy.png
+
+## Resize images down to width, height, preserving aspect ratio.
+## Nook size, with margin.
+image_max_size: 580, 725
+
+## Change image to grayscale, if graphics library allows, to save
+## space.
+#grayscale_images: false
+
+## if the <img> tag doesn't have a div or a p around it, nook gets
+## confused and displays it on every page after that under the text
+## for the rest of the chapter.  I doubt adding a div around the img
+## will break any other readers, but in case it does, the fix can be
+## turned off.
+nook_img_fix:true
+
 [mobi]
 ## mobi TOC cannot be turned off right now.
 #include_tocpage: true

-
 ## Each site has a section that overrides [defaults] *and* the format
 ## sections test1.com specifically is not a real story site.  Instead,
 ## it is a fake site for testing configuration and output.  It uses
@@ -234,8 +292,92 @@ extratags: FanFiction,Testing,Text
 [test1.com:html]
 extratags: FanFiction,Testing,HTML

+[castlefans.org]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+[fanfiction.mugglenet.com]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+[fanfiction.portkey.org]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[fanfiction.tenhawkpresents.com]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+[nfacommunity.com]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+[thequidditchpitch.org]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.adastrafanfic.com]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.archiveofourown.org]
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
 [www.fanfiction.net]

+[www.ficbook.net]
+
+[www.fictionalley.org]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
 [www.fictionpress.com]
 ## Clear FanFiction from defaults, fictionpress.com is original fiction.
 extratags:
@@ -248,6 +390,91 @@ extratags:
 #username:YourName
 #password:yourpassword

+[www.fimfiction.net]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+## fimfiction.net stories can be locked requiring individual
+## passwords.  If fail_on_password is set, the downloader will fail
+## when a password is required rather than prompting every time.
+#fail_on_password: false
+
+[www.harrypotterfanfiction.com]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.hpfandom.net]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.libraryofmoria.com]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+[www.mediaminer.org]
+
+[www.midnightwhispers.ca]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+## some sites include images that we don't ever want becoming the
+## cover image.  This lets you exclude them.
+cover_exclusion_regexp:/stories/999/images/.*?_trophy.png
+
+[www.potionsandsnitches.net]
+
+[www.siye.co.uk]
+
+[www.squidge.org/peja]
+# www.squidge.org/peja calls it Fandom <shrug>
+category_label:Fandom
+# Remove numWords -- www.squidge.org/peja word counts are inaccurate
+titlepage_entries: series,category,genre,language,characters,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,site,description
+
+[www.squidge.org/peja:txt]
+## Add URLs since there aren't links.
+# Remove numWords -- www.squidge.org/peja word counts are inaccurate
+titlepage_entries: series,category,genre,language,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,site,storyUrl, authorUrl, description
+
+[www.thewriterscoffeeshop.com]
+## Some sites require login (or login for some rated stories) The
+## program can prompt you, or you can save it in config.  In
+## commandline version, this should go in your personal.ini, not
+## defaults.ini.
+#username:YourName
+#password:yourpassword
+
+## Some sites also require the user to confirm they are adult for
+## adult content.  In commandline version, this should go in your
+## personal.ini, not defaults.ini.
+#is_adult:true
+
+## thewriterscoffeeshop.com (ab)uses series as personal reading lists.
+collect_series: false
+
+[www.tthfanfic.org]
+## Some sites do not require a login, but do require the user to
+## confirm they are adult for adult content.  In commandline version,
+## this should go in your personal.ini, not defaults.ini.
+#is_adult:true
+
+## tth is a little unusual--it doesn't require user/pass, but the site
+## keeps track of which chapters you've read and won't send another
+## update until it thinks you're up to date.  This way, on download,
+## it thinks you're up to date.
+#username:YourName
+#password:yourpassword
+
 [www.twilighted.net]
 ## Some sites require login (or login for some rated stories) The
 ## program can prompt you, or you can save it in config.  In
@@ -272,68 +499,12 @@ collect_series: false

 [www.whofic.com]

-[www.mediaminer.org]
-
-[www.thewriterscoffeeshop.com]
-## Some sites require login (or login for some rated stories) The
-## program can prompt you, or you can save it in config.  In
-## commandline version, this should go in your personal.ini, not
-## defaults.ini.
-#username:YourName
-#password:yourpassword
-
-## Some sites also require the user to confirm they are adult for
-## adult content.  In commandline version, this should go in your
-## personal.ini, not defaults.ini.
-#is_adult:true
-
-## thewriterscoffeeshop.com (ab)uses series as personal reading lists.
-collect_series: false
-
-[www.ficwad.com]
-## Some sites require login (or login for some rated stories) The
-## program can prompt you, or you can save it in config.  In
-## commandline version, this should go in your personal.ini, not
-## defaults.ini.
-#username:YourName
-#password:yourpassword
-
-[www.adastrafanfic.com]
+[www.wraithbait.com]
 ## Some sites do not require a login, but do require the user to
 ## confirm they are adult for adult content.  In commandline version,
 ## this should go in your personal.ini, not defaults.ini.
 #is_adult:true

-[www.fictionalley.org]
-## Some sites do not require a login, but do require the user to
-## confirm they are adult for adult content.  In commandline version,
-## this should go in your personal.ini, not defaults.ini.
-#is_adult:true
-
-[www.harrypotterfanfiction.com]
-## Some sites do not require a login, but do require the user to
-## confirm they are adult for adult content.  In commandline version,
-## this should go in your personal.ini, not defaults.ini.
-#is_adult:true
-
-[www.fimfiction.net]
-## Some sites do not require a login, but do require the user to
-## confirm they are adult for adult content.  In commandline version,
-## this should go in your personal.ini, not defaults.ini.
-#is_adult:true
-
-[www.tthfanfic.org]
-## Some sites do not require a login, but do require the user to
-## confirm they are adult for adult content.  In commandline version,
-## this should go in your personal.ini, not defaults.ini.
-#is_adult:true
-## tth is a little unusual--it doesn't require user/pass, but the site
-## keeps track of which chapters you've read and won't send another
-## update until it thinks you're up to date.  This way, on download,
-## it thinks you're up to date.
-#username:YourName
-#password:yourpassword
-
 [overrides]
 ## It may sometimes be useful to override all of the specific format,
 ## site and site:format sections in your private configuration.  For
@@ -7,6 +7,28 @@
 ## personal.ini, not defaults.ini.
 #is_adult:true

+## include images from img tags in the body and summary of
+## stories.  Images will be converted to jpg for size if possible.
+#include_images:true
+
+## If not set, the summary will have all html stripped for safety.
+## Both this and include_images must be true to get images in the
+## summary.
+#keep_summary_html:true
+
+## If set, the first image found will be made the cover image.  If
+## keep_summary_html is true, any images in summary will be before any
+## in chapters.
+#make_firstimage_cover:true
+
+## Resize images down to width, height, preserving aspect ratio.
+## Nook size, with margin.
+#image_max_size: 580, 725
+
+## Change image to grayscale, if graphics library allows, to save
+## space.
+#grayscale_images: false
+
 ## Try to collect series name and number of this story in series.
 ## Some sites (ab)use 'series' for reading lists and personal
 ## collections.  This lets us turn it on and off by site without
@@ -55,6 +77,7 @@

 [www.fimfiction.net]
 #is_adult:true
+#fail_on_password: false

 [www.tthfanfic.org]
 #is_adult:true
@@ -10,5 +10,10 @@ python downloader.py http://www.fanfiction.net/s/5192986/1/A_Fox_in_Tokyo

 Do 'python downloader.py -h' for more options.

-This tool uses Python 2.5.2, but should work with newer versions of Python.
+This tool uses Python 2.7, but should work with newer versions of Python.

+For more information, see:
+
+http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderSupportedsites
+
+http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderFAQs
Author	SHA1	Message	Date
Jim Miller	5072572f1c	Plugin only, optionally search epub text for story URL, option to GC only new.	2012-05-18 12:11:16 -05:00
Jim Miller	c4bbf3f072	Added tag FanFictionDownLoader-4.4.9 for changeset 5b585e031b72	2012-05-12 18:49:02 -05:00
Jim Miller	6413c52584	Added tag calibre-plugin-1.5.20 for changeset 5b585e031b72	2012-05-12 18:48:40 -05:00
Jim Miller	37bc1306c9	Integrate new adapters www.wraithbait.com and www.libraryofmoria.com.	2012-05-12 18:47:22 -05:00
Ida	7f8c39227f	Fixed dates and rating for WraithBait.com	2012-05-12 18:44:40 -04:00
Jim Miller	d4988545b5	Merge	2012-05-12 16:00:28 -05:00
Jim Miller	5ceb96c913	Adding WWOMB(www.squidge.org/peja) adapter. Some infrastructure changes to allow config section [www.squidge.org/peja]. More needed to support other sites under www.squidge.org.	2012-05-12 15:58:51 -05:00
Ida	3d735e430d	First version of Libraryofmoria.com and Wraithbait.com adapters	2012-05-10 17:38:12 -04:00
Jim Miller	3ccb895c7b	Added tag calibre-plugin-1.5.18 for changeset 3beaa4617bd3	2012-05-08 10:43:26 -05:00
Jim Miller	f0a1e1d471	Plugin--Add feature to integrate with Generate Cover plugin by kiwidude.	2012-05-08 10:43:02 -05:00
Jim Miller	3719e9860c	Added tag CLI-4.4.8a for changeset d38c36148dc4	2012-04-25 21:53:28 -05:00
Jim Miller	ea43dffe0c	Added tag calibre-plugin-1.5.16 for changeset d38c36148dc4	2012-04-25 21:53:09 -05:00
Jim Miller	0a2cff3469	Yet more fixes for poor parsing of numeric entities--this time, my poor parsing.	2012-04-25 21:52:56 -05:00
Jim Miller	3cb92f48cc	Added tag FanFictionDownLoader-4.4.8 for changeset 8fc7a6cc2d87	2012-04-22 10:10:08 -05:00
Jim Miller	e5cc4ebbd5	Bump Web/CLI version.	2012-04-22 10:09:44 -05:00
Jim Miller	c6c67f99ca	Moved tag calibre-plugin-1.5.15 to changeset a1a840bcd4c9 (from changeset bac4028b803d)	2012-04-22 09:55:58 -05:00
Jim Miller	c75e62d07e	Merge fimfiction changes to trunk.	2012-04-22 09:55:39 -05:00
Jim Miller	e9b203eb57	Added tag calibre-plugin-1.5.15 for changeset bac4028b803d	2012-04-21 21:37:27 -05:00
Jim Miller	a2eaa9fb22	Fix for ArchiveOfOurOwn site change.	2012-04-21 21:37:16 -05:00
althaine	0cf5ec9880	Updated fimfic.net adapter to use provided API	2012-04-20 20:36:20 +10:00
Jim Miller	465e25d3e6	Added tag FanFictionDownLoader-4.4.7 for changeset c8ab1f6bbd57	2012-04-19 12:19:06 -05:00
Jim Miller	0c85cf1d10	Web- Add list of all downloads feature to web service.	2012-04-19 12:18:40 -05:00
Jim Miller	7f7a1a983c	Plugin-Fix setting status column with no status available(fictionalley).	2012-04-19 12:14:51 -05:00
Jim Miller	6f4be261ef	Fix Japanese in CLI & web version. Broke it in 443 and didn't notice.	2012-04-18 17:54:42 -05:00
Jim Miller	04eb551d48	Added tag CLI-4.4.6 for changeset b69d6004ae25	2012-04-17 14:07:50 -05:00
Jim Miller	6299fc7582	Added tag calibre-plugin-1.5.14 for changeset b69d6004ae25	2012-04-17 14:07:29 -05:00
Jim Miller	c2d7b4ecce	Change plugin 'images' checkbox to add to [epubs], kludge for fimfiction images	2012-04-17 14:07:19 -05:00
Jim Miller	01b14701a0	Moved tag calibre-plugin-1.5.13 to changeset b55fabb0b8c9 (from changeset b0720fc31c10)	2012-04-11 11:11:43 -05:00
Jim Miller	08aa677855	Fix for Python GC'ing menu objects with no explicit Python references.	2012-04-11 11:11:25 -05:00
Jim Miller	bd477d2696	Added tag FanFictionDownLoader-4.4.5 for changeset b0720fc31c10	2012-04-10 09:43:20 -05:00
Jim Miller	1f3220a221	Added tag calibre-plugin-1.5.13 for changeset b0720fc31c10	2012-04-10 09:42:32 -05:00
Jim Miller	b4a8361df9	Package Ida's new archive.skyehawke.com adapter for release.	2012-04-10 09:42:01 -05:00
Ida	e647b0e69b	First version adapter for archive.skyehawke.com	2012-04-10 01:59:34 -04:00
Jim Miller	a5fd8646b7	Added tag calibre-plugin-1.5.12 for changeset 6145294332cd	2012-04-09 11:35:25 -05:00
Jim Miller	ce8cf44101	Set calibre's author link field to authorUrl.	2012-04-09 11:34:52 -05:00
Jim Miller	d5f1484cb7	Added tag calibre-plugin-1.5.11 for changeset 5ad254071cc4	2012-03-24 12:09:31 -05:00
Jim Miller	ad092e2d5f	Another work around for SGMLParser's entity handling. &#8212again v &#8212a;gain	2012-03-24 12:09:18 -05:00
Jim Miller	a9e873490d	Added tag FanFictionDownLoader-4.4.4 for changeset 4a7cf5c4919c	2012-03-19 21:06:20 -05:00
Jim Miller	53c80e3c71	Added tag calibre-plugin-1.5.10 for changeset 4a7cf5c4919c	2012-03-19 21:06:10 -05:00
Jim Miller	f32667ea04	Update index.html, plugin-defaults.ini, bump versions.	2012-03-19 21:06:00 -05:00
Jim Miller	7c43ecb56d	Make default_cover_image use ${title}, etc; add cover_exclusion_regexp feature.	2012-03-19 16:22:51 -05:00
Jim Miller	3a8d045e7a	Add ksarchive.com adapter from Jade Aislin.	2012-03-19 16:16:18 -05:00
Jim Miller	55b9f90c66	Fixes for mugglenet.com--add user/pass, better warning & summary handling.	2012-03-19 16:15:09 -05:00
Jim Miller	22be02369f	Added tag FanFictionDownLoader-4.4.3 for changeset 361de9f22e6e	2012-03-16 17:23:29 -05:00
Jim Miller	392a5e834c	Added tag calibre-plugin-1.5.9 for changeset 361de9f22e6e	2012-03-16 17:23:18 -05:00
Jim Miller	f2794b1e64	Bump web/cli version, update index.html.	2012-03-16 17:23:08 -05:00
Jim Miller	0f68c6ec29	Bump plugin version, add links to FAQ & Supported list to About & Config. Add DB error to mw.	2012-03-16 17:02:58 -05:00
Jim Miller	8c5f784afb	Remove gayauthors.org at the site admin's request.	2012-03-16 12:27:06 -05:00
Jim Miller	f207ba01fe	Adding Sam's code for nfacommunity.com & midnightwhispers.ca.	2012-03-16 12:26:45 -05:00
Jim Miller	59d0040e64	Make 'Update Calibre Only' add a new book if no matching book found.	2012-03-14 16:51:41 -05:00
Jim Miller	949de85150	Turn off gayauthors.org. Wyndham reports complaints from site admin.	2012-03-13 17:21:20 -05:00
Jim Miller	1d1494d11c	Turn off gayauthors.org. Wyndham reports complaints from site admin.	2012-03-13 17:12:43 -05:00
Jim Miller	1d474f6493	Improve image support for a couple of obscure cases.	2012-03-13 14:08:40 -05:00
Jim Miller	b31b497698	portkeyorg- Write 'Chapter does not exist' chapter when chapter does not exist.	2012-03-12 21:43:29 -05:00
Jim Miller	eac6304e98	Add is_adult check to twiwrite for when user on site is not set to adult.	2012-03-12 21:42:39 -05:00
Jim Miller	df3655a57c	Change td in portkey chapter text to div for nook.	2012-03-07 15:22:21 -06:00
Jim Miller	b7a3f327b0	Minor improvement to column updates for new stories.	2012-03-07 10:48:38 -06:00
Jim Miller	42fa7cfc3e	Moved tag FanFictionDownLoader-4.4.2 to changeset 3f66d81535fa (from changeset 5df1a1ed38c9)	2012-03-06 21:12:10 -06:00
Jim Miller	9fb319f42f	Moved tag calibre-plugin-1.5.8 to changeset 3f66d81535fa (from changeset 01d1a20329b9)	2012-03-06 21:11:42 -06:00
Jim Miller	bcd2f2f9ec	Don't mark TtH BtVS when Non-Crossover, unless BtVS/Angel Non-Cross.	2012-03-06 21:11:27 -06:00
Jim Miller	9855a6adec	Moved tag FanFictionDownLoader-4.4.2 to changeset 5df1a1ed38c9 (from changeset 01d1a20329b9)	2012-03-06 16:56:56 -06:00
Jim Miller	be3b08116c	Allow but not require www. on thequidditchpitch.org.	2012-03-06 16:56:34 -06:00
Jim Miller	763b52d05d	Change previous version link in index.html.	2012-03-06 10:24:23 -06:00
Jim Miller	952bec41cd	Added tag FanFictionDownLoader-4.4.2 for changeset 01d1a20329b9	2012-03-05 22:56:01 -06:00
Jim Miller	1ac285cbee	Added tag calibre-plugin-1.5.8 for changeset 01d1a20329b9	2012-03-05 22:50:25 -06:00
Jim Miller	122022ce88	Add thequidditchpitch.org, clean up comments, housekeeping for new sites.	2012-03-05 22:50:05 -06:00
Jim Miller	1f7ad4fb71	Add hpfandom.net.	2012-03-05 21:12:46 -06:00
Jim Miller	35fea093b8	Add mugglenet.com.	2012-03-05 17:37:17 -06:00
Jim Miller	e9a6f8612d	Really add portkey.org.	2012-03-05 14:22:50 -06:00
Jim Miller	f32907b070	Add portkey.org.	2012-03-05 14:22:30 -06:00
Jim Miller	a5916f8d97	Added tag CLI-4.4.1 for changeset 4ba059184bce	2012-03-02 12:11:08 -06:00
Jim Miller	ce2de180d2	Fix for pre-existing epubs with stripped img tags without breaking images.	2012-03-01 21:50:32 -06:00
Jim Miller	0917460e68	Fix for pre-existing epubs with stripped img tags.	2012-03-01 21:27:48 -06:00
Jim Miller	3bb26c2b58	Merge images branch back into trunk(default).	2012-03-01 19:38:51 -06:00
Jim Miller	9739706c8f	Added tag FanFictionDownLoader-4.4.0 for changeset 9a70744bc1cc	2012-03-01 19:32:02 -06:00
Jim Miller	753ffef8bc	Bump web service/CLI version, too, update announcements.	2012-03-01 19:31:22 -06:00
Jim Miller	652f9c7bc9	Added tag calibre-plugin-1.5.7 for changeset 3b1d63100f1f	2012-03-01 18:02:30 -06:00
Jim Miller	4802b5ef60	Plugin v1.5.7.	2012-03-01 18:02:24 -06:00
Jim Miller	2995941c21	Add nook_img_fix--Nook show img under text on later pages if it doesn't have a div or p around it. Wraps img tags in a div, defaults true.	2012-03-01 18:00:35 -06:00
Jim Miller	4e261a7f39	Added tag calibre-plugin-1.5.6 for changeset caffd21c5c2d	2012-03-01 13:53:40 -06:00
Jim Miller	8365b481c3	Add 'Include images in EPUBs' as checkbox option for visibility to casual users.	2012-02-29 22:17:53 -06:00
Jim Miller	a5ce4d7476	Added tag calibre-plugin-1.5.5 for changeset d0729d7f2e57	2012-02-29 20:11:13 -06:00
Jim Miller	8d2a1341ce	Support for fimfiction.net's story passwords and fimfic fail_on_password option.	2012-02-29 20:11:05 -06:00
Jim Miller	9b1b415fec	Added tag calibre-plugin-1.5.4 for changeset 854e4ee3f736	2012-02-29 17:23:10 -06:00
Jim Miller	54f5d53f33	Fixes for FimFic--missing story, story password, updated date.	2012-02-29 17:23:02 -06:00
Jim Miller	d3dc23b950	Added tag calibre-plugin-1.5.3 for changeset 2f454010da02	2012-02-29 12:44:23 -06:00
Jim Miller	189832a7a9	Add replace_hr & never_make_cover options, allow empty fimfiction stories.	2012-02-29 12:44:12 -06:00
Jim Miller	6528b3c9a5	Added tag calibre-plugin-1.5.2 for changeset d8ff11416ae5	2012-02-28 14:27:43 -06:00
Jim Miller	dd293909c0	GA fails on some chapters. Soup seems to work better than Stone this time.	2012-02-28 14:27:32 -06:00
Jim Miller	27ec7228ae	Added tag calibre-plugin-1.5.1 for changeset 4abbb7f1efb4	2012-02-28 13:27:10 -06:00
Jim Miller	d8bf424270	Handle image errors, tweak desc of Update Cover option.	2012-02-28 13:27:02 -06:00
Jim Miller	5a861cafc6	Add 'stay on top' option for add dialog. Fix support for updating TtH epubs.	2012-02-27 22:07:07 -06:00
Jim Miller	0700104d7e	Added tag calibre-plugin-1.5.0 for changeset 7969ed0c65e2	2012-02-27 12:25:17 -06:00
Jim Miller	6a525ca9fb	Change origsrc attribute on img to longdesc to appease epubcheck. Clean up.	2012-02-27 12:25:03 -06:00
Jim Miller	4306bfc301	Merge changes from trunk.	2012-02-27 10:52:49 -06:00
Jim Miller	e4ce51787c	Added tag CLI-4.3.3 for changeset 5999c9b09760	2012-02-27 10:48:25 -06:00
Jim Miller	a9cecbbe4f	Rewrite of epub update, now handles images.	2012-02-26 20:32:19 -06:00
Jim Miller	f0445f106c	First version of cover image support. Needs more work--update merge is broken.	2012-02-26 15:19:41 -06:00
Jim Miller	a89cec6844	Image resize and grayscale support for CLI when PIL is installed. CLI warning.	2012-02-25 16:14:27 -06:00
Jim Miller	baaf151fa2	Hardcode not to d/l images in appengine. Better no-img lib support for non-jpg. Pass formtype into adapter in CLI & Web. Center fimfic cover image in summary.	2012-02-24 22:24:55 -06:00
Jim Miller	0bea4afd01	Update ga for image support changes. Change img file names. Possibility of dups between updates, but not bad overwrites or overlength names,	2012-02-24 20:50:43 -06:00
Jim Miller	6a83131a99	Merging changes from trunk.	2012-02-24 18:42:52 -06:00
Jim Miller	076744f2d4	Save changes to defaults.	2012-02-24 18:41:19 -06:00
Jim Miller	668946adcc	Change app.yaml back.	2012-02-24 16:40:27 -06:00
Jim Miller	b10dca9ae5	Added tag calibre-plugin-1.4.6 for changeset af1de3b15349	2012-02-24 16:39:47 -06:00
Jim Miller	3f9abb8c07	Activate gayauthors.org, support ga authors with space, bump plugin version.	2012-02-24 16:39:37 -06:00
Ida	1274f37668	Fixed authors for gayauthors.net and removed unnecesary stuff from AO3	2012-02-24 01:22:59 -05:00
Ida	99891ae176	v1 of the adapter. Doesn't work for authors that have a space in their name...	2012-02-23 23:25:49 -05:00
Jim Miller	35bb2a49ba	Merging change from trunk.	2012-02-23 20:33:25 -06:00
Jim Miller	f0006a7db7	Moving image resize code up to when img is collected rather than written.	2012-02-23 20:32:24 -06:00
Jim Miller	e1e7a79006	Change to url drop/paste to stop doubled URLs from Chrome.	2012-02-23 20:28:47 -06:00
Jim Miller	ba9b82657a	Adding support for images in chapters and description.	2012-02-23 14:13:22 -06:00
Jim Miller	71e33ed8ea	Added tag calibre-plugin-1.4.5 for changeset 0157333ee07b	2012-02-22 10:43:40 -06:00
Jim Miller	640c676cc5	Change web index.html message.	2012-02-22 10:42:21 -06:00
Jim Miller	f29844fe14	fanfiction.net is added CR characters sometimes now. Bump plugin version.	2012-02-22 10:39:05 -06:00
Jim Miller	b6c0917bcc	Added tag calibre-plugin-1.4.4 for changeset 86e8f489854b	2012-02-21 14:40:06 -06:00