votorola.a.diff.harvest

Class PipermailHarvester

java.lang.Object

PipermailHarvester

public class PipermailHarvester
extends Object

A harvester implementation for pipermail archives. Pipermail archives are generated by mailman. Harvested messages are stored in HarvestCache and fetchers are scheduled with HarvestRunner.

Basically we do a depth first search sorted by date. Web-view on the remote archive for this harvester:

 Tree of linked pages.     Level      Example remote archive URLs
             r1              root       http://mail.zelea.com/list/votorola/    
            / \
           /   \
          /     \
         /       \
       i2        i3          index      2010-Jan/date.html                      
       / \       /|\
      /   \     / | \
     m4    m5 m6 m7 m8       message    2010-Jan/003321.html
 

All fetchers depend on one HTTP request. Runtime steps for these Fetchfetchers:

1. Create fetcher and schedule it
2. Once HarvestRunner handles the fetcher, it asynchronously fetches the HTTP remote page and runs the fetcher in its thread pool
3. Having access to AbstractFetcher.getInputStream() now, the fetcher parses the page
4. r1, i2, i3 start fetchers for the index of URLs parsed out of the page

Result after r1: i2-i3, after i2: m4-m5-i3 after i3: m6-m7-m8.
The state is save by anonymously extending the last Messagefetcher like m5 or m8 appropriately.

Note: This is only an example. You can submit your Fetcher differently and you can also save state differently. Usage of the HarvestRunner is recommended for graceful I/O handling though.

See Also:

The communication diagram for details of the overall harvesting concept.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) class	PipermailHarvester.Archive Structure to save state of a single archive.
(package private) class	PipermailHarvester.ContextFetcher Provides update context.
(package private) class	PipermailHarvester.MessageFetcher A fetcher for a single page, like "2010-Jan/003882.html".
(package private) class	PipermailHarvester.MyKickReceiver Receives kicks from Kicker for this Harvester.
(package private) class	PipermailHarvester.PeriodFetcher An index fetcher for one period like "2010-January/date.html" or "2012/date.html".
(package private) class	PipermailHarvester.RootFetcher An index fetcher for the whole pipermail archive, like "http://mail.zelea.com/list/votorola/".
(package private) class	PipermailHarvester.UpdateContext Context to run fetchers and synchronize state during an update.

Field Summary

Fields

Modifier and Type	Field and Description
(package private) Map<String,PipermailHarvester.Archive>	archives Tracks all configured archives.
static Pattern	PAT_AUTHOR Pattern to find author email in PipermailHarvester.MessageFetcher.
static Pattern	PAT_AUTHOR2 Pattern to parse author email in PipermailHarvester.MessageFetcher.
static Pattern	PAT_CONTENT_END Pattern to find the end of the body of the message in PipermailHarvester.MessageFetcher.
static Pattern	PAT_CONTENT_START Pattern to find the start of the body of the message in PipermailHarvester.MessageFetcher.
static Pattern	PAT_INPUTENC Find input encoding.
static Pattern	PAT_LANG Parse language.
static Pattern	PAT_LISTINFO Pattern for list-info page.
static Pattern	PAT_PERIOD Pattern to parse sub-list of posts for each month in PipermailHarvester.RootFetcher.
static Pattern	PAT_POST Pattern to parse list of posts in this month list in PipermailHarvester.PeriodFetcher.
static Pattern	PAT_SENTDATE Pattern to parse sent date in PipermailHarvester.MessageFetcher.

Constructor Summary

Constructors

Constructor and Description
PipermailHarvester() Construct a new harvester for pipermail.

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PAT_LISTINFO

public static final Pattern PAT_LISTINFO

Pattern for list-info page. (for setup)

PAT_INPUTENC

public static final Pattern PAT_INPUTENC

Find input encoding.

PAT_LANG

public static final Pattern PAT_LANG

Parse language.

PAT_PERIOD

public static final Pattern PAT_PERIOD

Pattern to parse sub-list of posts for each month in PipermailHarvester.RootFetcher. If period is in years: 2012/date.html ... in months: 2012-October/date.html TODO add/fix weekly pattern if used somewhere

PAT_POST

public static final Pattern PAT_POST

Pattern to parse list of posts in this month list in PipermailHarvester.PeriodFetcher.

PAT_AUTHOR

public static final Pattern PAT_AUTHOR

Pattern to find author email in PipermailHarvester.MessageFetcher.

PAT_AUTHOR2

public static final Pattern PAT_AUTHOR2

Pattern to parse author email in PipermailHarvester.MessageFetcher.

PAT_SENTDATE

public static final Pattern PAT_SENTDATE

Pattern to parse sent date in PipermailHarvester.MessageFetcher.

PAT_CONTENT_START

public static final Pattern PAT_CONTENT_START

Pattern to find the start of the body of the message in PipermailHarvester.MessageFetcher.

PAT_CONTENT_END

public static final Pattern PAT_CONTENT_END

Pattern to find the end of the body of the message in PipermailHarvester.MessageFetcher.

archives

final Map<String,PipermailHarvester.Archive> archives

Tracks all configured archives. Key is the base URL of the archive. declared package private for example documentation

Constructor Detail

PipermailHarvester

public PipermailHarvester()

Construct a new harvester for pipermail.