retiefjimm
f584b3f1e8
Change calls for str() to unicode(). Default CL type to epub if not given.
2010-11-21 11:52:11 -06:00
retiefjimm
df6b85b723
Fix desc escaping. <sigh>
2010-11-18 23:34:25 -06:00
wsuetholz
26f6ddabd9
The with call is not an automatic in python 2.5, you need to import with_statement for it to work.
...
the from __future__ import with_statement needs to be the first import statment as well.
2010-11-18 20:08:55 -06:00
wsuetholz
22d0989d0a
Raise exceptions instead of an exit when errors are detected.
...
Added static functions to the writer classes that contain the type name and type extension to use when writing out the destination file.
Added a function to the zipdir that will check the date the fic was last updated in the current archive of a fic against the current last updated date and return whether the fic is newer then the archive file. This is not useful at this time for when this is running under appEngine.
Added a flag to indicate if the program is running from appEngine or not.
2010-11-18 16:07:58 -06:00
retiefjimm
9b6fdb4540
Fix numeric entities--correctly this time. Reduce margins slightly, tweak description escaping slightly.
2010-11-18 14:26:16 -06:00
wsuetholz
e3b4769bfb
Misspelled the return variable from getOutputFileName.
2010-11-16 09:09:26 -06:00
wsuetholz
5618d8036b
MediaMiner was not sending the date with the correct timezone info, so strip it off. We don't need the time anyways.
2010-11-13 23:30:47 -06:00
wsuetholz
dbec6c95b2
Changed to deal with somebody putting in the mobile url for fanfiction.net... ie < http://m.fanfiction.net/s/11 > will get changed to < http://www.fanfiction.net/s/11 >
2010-11-12 21:51:46 -06:00
wsuetholz
c77498e6ea
Some changes in order to allow the refactoring of functions to work from within the appEngine.
2010-11-12 21:28:25 -06:00
wsuetholz
99ba1b48aa
The getSubjects function was returning the author.
2010-11-12 20:27:17 -06:00
wsuetholz
bdf429b41c
Added in mediaminer.org adapter. Still has problems becaue mm seems to use the ≴ type character encoding, and I couldn't manage to replace them.
2010-11-12 20:25:55 -06:00
wsuetholz
981a922d00
Deal with being passed in the first chapter as the url.
2010-11-11 13:53:48 -06:00
wsuetholz
b51dd549f5
self.numChapters was one too many.
2010-11-11 13:22:30 -06:00
wsuetholz
35065593af
Refactored source code to remove redundant code.. Moved most of the adapter get and set functions for variables into the adapter.py. Define all the adapter variables in adapter.py. Remove the unneeded functions from the various adapter sources.
...
Fix a couple of problems with fpcom.py. Rearrange the Genre and Category for some of the adapters.
2010-11-11 13:14:15 -06:00
wsuetholz
696223a456
Added in an adapter for FictionPress.com. This started out as a copy of ffnet.py, but the extractIndividualUrls rapidly became completly different. I have not found a field that says total number of words for the story.
2010-11-11 09:55:37 -06:00
wsuetholz
30b914788c
Add a new adapter for the potionsandsnitches web site. Once we started looking for the additional information for the title page, things were sufficiently different from the twilighted adapter that it couldn't be used anymore.
2010-11-10 23:11:08 -06:00
wsuetholz
ed4ff4b6ab
Trying to fix outputting some utf-8 text. Added the utf-8 header to all the source code.
...
Ended up modifying the removeEntities function to do a weird decode/encode step on the text passed in. This seems to at least stop things from crashing..
2010-11-10 13:47:13 -06:00
wsuetholz
3f40df82dd
Change to allow a chapter url to be passed in, yet still get the information it needs from the story index page. This will get the chapter links from the passed in url.
2010-11-10 12:36:21 -06:00
wsuetholz
7c252e83a1
It seems that self.numWords and self.numChapters can on occasion be numeric instead of string, so use the str() function when building the string for the title page.
2010-11-10 12:34:25 -06:00
wsuetholz
d4c18fe1a8
Changed all references to downaloder.py to downloader.py.
2010-11-10 11:29:17 -06:00
wsuetholz
b2ea9c3e73
Changed the title page generation to allow for easily adding and removing entries from the table of information shown.
...
Modified ffnet.py to use the mangled story title if given instead of the name portion of the URL. As part of this, the Name and chapter number are now optional in the URL passed in.
2010-11-10 11:18:48 -06:00
wsuetholz
33d1fabd11
Changeset Tag Branch User Date Summary
...
53:c93e07566456 wsuetholz 2010-11-10 10:11 Ficwad wants you to login in order to view some stories.. They old ficwad.py got around that by starting with the first chapter instead of the story index page. Since I needed the story index page I had changed it to switch to that page, and then scrape the chapter information from there, which doesn't work if the chapters are blocked. While it still won't work if you pass in the URL for the story index page, I now switch back to the page that you passed in originally when looking for the chapters to download.. The one problem I have with this, is I change the self.url to the story index page, which should probably remain so that we have a consistent self.url even if the user starts with chapter 9 this time instead of chapter 1.
2010-11-10 10:18:46 -06:00
retiefjimm
c326aa47b0
Need to reverse sort entities list to get entities with ';' ahead of versions without. Like '"' and '"'. Otherwise '"' becomes '";'.
2010-11-09 23:02:33 -06:00
wsuetholz
d67668de23
Changes to the format of the title page. Tried to fix the description field on the title page and the contents.opf page.
2010-11-09 18:20:37 -06:00
wsuetholz
363c211401
Had left in a debugging statements that should have been removed..
2010-11-09 18:18:42 -06:00
wsuetholz
379efc34f1
Add a title page to the resulting EPUB file. This required scraping more information from the web pages in order to populate the new fields. This change includes a change to the way that the output.py uses the adapters. It now gets passed in the adapter and then calls functions from the adapter in order to retrieve the scraped information. This will make it easier down the road when adding more information, or even pictures.
2010-11-09 16:35:46 -06:00
retiefjimm
778deaea00
Replace the haphazardly collected set of html entities with a list of known (browser) supported entities. Will now preserve right/left quotes.
2010-10-28 17:29:00 -05:00
retiefjimm
33c3f0b4fd
Added tag fanficdownloader-0.2 for changeset 3eff15bceebe
2010-10-20 15:02:11 -05:00
retiefjimm
5bca3106df
Adding a twipassword.py and a quick readme.txt.
2010-10-20 14:58:48 -05:00
retiefjimm
c056ed0740
Fix zip routine so it checks for file mimetype doesn't just assume it exists.
2010-10-16 11:30:09 -05:00
retiefjimm
25cbab225c
potionsandsnitches.net doesn't require login. Indeed, it doesn't even have the same login URL.
2010-10-14 12:42:29 -05:00
retiefjimm
4e53481d7d
Somewhat kludgey fix for the kludgey way we extract the story text from fictionalley for stories with poor html.
2010-10-13 00:01:24 -05:00
retiefjimm
b9a72d372d
Add � entity support, fix "<authorname> aa" from testing, add potionsandsnitches.net.
2010-10-11 20:44:19 -05:00
retiefjimm
7f8d7a7dbb
Add '\n' after each </p> and <br /> tag. Extremely long(200k+) single line chapters (like ffnet gives) causes problems for nook.
...
Also removed a little dead code from ffnet.py.
2010-10-11 14:05:57 -05:00
retiefjimm
dafa33c64e
Generalize handling of &#<number>; entities. Generalize method to create a valid filename.
...
Remove print from fictionalley.py that was causing the output-before-302-header issue.
Some changes to chapter URL formation due to apparent differences in handling '//' in URLs in appengine vs downalod.py in 2.5.2.
2010-10-10 12:54:41 -05:00
retiefjimm
275a1dd6ad
Add '->'\'' to entity translation list.
2010-10-08 21:54:35 -05:00
retiefjimm
f8701a9e8d
Fix to force mimetype file to be uncompressed and first in epub. Tested on Python 2.5.2.
2010-10-08 18:46:03 -05:00
retiefjimm
466d5f4ee3
Remove Entities, fix bar ampersands in story & author names.
2010-10-06 14:31:21 -05:00
retiefjimm
ca4b09d69d
Fixes for some obscure encoding errors. UTF8 versions of apostrophe and dash in story titles & chapter titles could cause problems in a few places.
2010-10-03 19:23:39 -05:00
retiefjimm
f75910ce7d
Stop doing prettify on the HTML, it sometimes introduces extraneous white space. Change the chapter file names and ids inside the metadata to sequential chapter0001 rather than base64 of the chapter title. Replace invalid center & u tags with div styled centered and span styled underlined. Remove dead ffa.py file. Change downloda.py to not call ffa.py, but do call fictionalley.
2010-09-30 21:33:58 -05:00
retiefjimm
773ff3c03c
Improved metadata for epubcheck, fixed a problem I introduced with >/<.
2010-09-29 21:32:04 -05:00
retiefjimm
9434431c7b
Add twilighted.net(not com) to downaloder.py so it actually *works*. Add one-shot support for twilighted.net.
2010-09-28 19:52:30 -05:00
retiefjimm
3be9b3331e
Replace &#XXX; entities in file names, allow BR tags--some stories use multiple BRs are section breaks.
2010-09-28 19:26:19 -05:00
retiefjimm
84fea5b896
Fictionalley.com working, also add twilighted.com to downaloder.py.
2010-09-28 19:26:06 -05:00
retiefjimm
fe67201f95
Fix up code for harrypotterfanfiction.com so it works and add one-shot support. Add < and > to entities list.
2010-09-28 13:26:49 -05:00
retiefjimm
ccb3fb36c5
Allow '&' and '£'. Allow single chapter/one-shot stories from FicWad. Correct story names from FicWad. Stop creating empty 'Story Index' chapter in stories from FicWad. Exit on chapter d/l error from FicWad.
2010-09-26 19:08:49 -05:00
Jimm
31c68ff796
Exit on ff.net error downloading chapter. Replace ':' with '_' in file name for html output, too.
2010-09-25 11:55:31 -05:00
retiefjimm
950807747c
Replace ':' in filename with '_' for Windoze. appspot version appears to already do this.
2010-09-25 00:44:56 -05:00
retiefjimm
b22d961e73
Allow one-shot fics for ffnet.
...
Write epub files as mode 'b' binary so it works on Windows.
Allow hr tags in epub output.
Don't compress file mimetype in epub file--required by standard.
2010-09-24 23:36:05 -05:00
sigizmund
2b2e7d52ec
Starting to use source control again after a long break :-)
2010-07-17 14:44:14 +01:00
retiefjimm