You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Doğacan Güney <do...@gmail.com> on 2010/07/03 12:01:18 UTC
Nutchbase design doc
Hello everyone,
I am attaching first draft of a complete nutchbase design document. There
are parts missing and parts not yet explained clearly but I would like to
get everyone's opinion on what they think so far. Please let me
know which parts are unclear, which parts make no sense etc, and I will
improve the draft.
-----------------------------------------------------------------------------------------------------------------------------------------
Nutchbase
=========
1) Rationale
* All your data in a central location (at least, nutch gives you the
illusion of a centralized storage)
* No more segment/crawldb/linkdb merges.
* No more "missing" data in a job. There are a lot of places where we copy
data from one structure to another just so that it is available in a later
job. For example, during parsing we don't have access to a URL's fetch
status. So we copy fetch status into content metadata. This will no longer
be necessary after nutchbase. When writing a job or a new plugin, programmer
only needs to specify which fields she wants to read and they will be
available to plugin / job.
* A much simpler data model. If you want to update a small part in a single
record, now you have to write a MR job that reads the relevant directory,
change the single record, remove old directory and rename new directory.
With nutchbase, you can just update that record.
2) Design
As mentioned above, all data for a URL is stored in a WebPage object. This
object is accessed by a key that is the reverse form of a URL. For example,
http://bar.foo.com:8983/to/index.html?a=b becomes
com.foo.bar:http:8983/to/index.html?a=b
If URLs are stored lexicographically, this means that URLs from same domain
and host are stored closer together. This will hopefully make developing
statistics tools easier for hosts and domains.
Writing a MapReduce job that uses Gora for storage does not take much
effort. There is a new class called StorageUtils that has a number of static
methods to make setting mappers/reducers/etc easier. Here is an example
(from GeneratorJob.java):
Job job = new NutchJob(getConf(), "generate: " + crawlId);
StorageUtils.initMapperJob(job, FIELDS, SelectorEntry.class,
WebPage.class,
GeneratorMapper.class, URLPartitioner.class);
StorageUtils.initReducerJob(job, GeneratorReducer.class);
An important argument is the second argument to #initMapperJob. This
specifies all the fields that this job will be reading. If plugins will run
during a job (for example, during ParserJob, several plugins will be
active), then before job is run, those plugins must be initialized and
FieldPluggable#getFields must be called for all plugins to figure out which
fields they want to read. During map or reduce phase, modifying WebPage
object is as simple as using the built-in setters. All changes will be
persisted.
Even though some of these objects are still there, most of the CrawlDatum,
Content, ParseData or similar objects are removed (or are slated to be
removed). In most cases, plugins will simply take (String key, WebPage page)
as arguments and modify WebPage object in-place.
3) Jobs
Nutchbase uses the concept of a marker to identify what has been processed
and what will be processed from now on. WebPage object contains a
map<String, String> called markers. For example, when GeneratorJob generates
a URL, it puts a unique string and a unique crawl id to this marker map.
Then FetcherJob only fetches a given URL if it contains this marker. At the
end of the crawl cycle, DbUpdaterJob clears all markers (except markers
placed by IndexerJob).
a) InjectorJob: This phase consists of two different jobs. First job reads
from a text file (as before) then puts a special inject marker. Second job
goes over all URLs then if it finds a WebPage with inject marker but nothing
else (so, it is a new URL), then this URL is injected (and marker is
deleted). Otherwise, marker is just deleted (since URL is already injected).
b) GeneratorJob: GeneratorJob is similar to what it was before, but it is
now a single job. During map phase:
if FetchSchedule indicates given URL is to be fetched then,
Calculate generator score using scoring filters
Output <SelectorEntry<URL, score>, WebPage>
SelectorEntry is sorted according to given score so highest scoring entries
will be processed first during reduce.
Then URLPartitioner partitions according to choices specified in config
files (by host, by ip or by domain).
Reduce phase counts URL according to topN and host/domain limits and marks
all URLs if limits are not yet reached.
GeneratorJob markes all generated URLs with a unique crawl id. Fetcher,
parser and indexer jobs can then use this crawl id to process only URLs that
are in that particular crawl cycle. Alternatively, they can also work on all
URLs (though, again, FetcherJob will only work on a URL which has been
marked by GenerateJob. So even if FetcherJob is instructed to work on "all"
URLs, it will skip those without generator markers).
GeneratorJob will also print crawl id to console.
c) FetcherJob: During map phase, FetcherJob outputs <Integer,
FetcherEntry<key, WebPage>> pairs. The first member of the pair (integer) is
a random integer between 0 and 65536. After map, these pairs are partitioned
into reduces according to URL's hosts. Reduce works exactly like old
Fetcher's map. FetcherJob can also continue interrupted fetches now (by
giving "-continue" switch on command line).
The random integer may seem pointless but it is actually quite important for
performance. Let's say instead of a random integer we were to just output
<key, WebPage> pairs from map. Again, let's say you have 1000 URLs from host
a.com, and 500 URLs from b.com. Let's also say that we have 10 fetcher
threads so maximum queue size is 500. In this case, since all URLs from host
a.com will be processed by reduce before all URLs from b.com, during reduce
phase, only one thread will fetch URLs from a.com while every other thread
will be spin-waiting. However, with randomization, URLs from a.com and
b.comwill be processed in a random order thus bandwidth utilization
will be
higher.
d) ParserJob: ParserJob is straightforward. It is only a map (i.e., has 0
reducers). Simple parses all URLs with active parse plugins.
e) DbUpdaterJob: This is a combination of updatedb and invertlinks jobs. If
a URL is successfully parsed (which means, it will contain a parse marker),
DbUpdaterJob will put its own marker. Note: It may make more sense to put a
marker even if a URL is not successfully parsed. DbUpdaterJob also cleans
all other markers.
f) IndexerJob: Goes over all URLs with a db update marker (again, you can
specify ALL URLs with update markers, or a crawl id), and indexes them.
4) What's missing
Most of the core functionality and plugins have been ported. However, some
tools and features are still missing: arc segment tools, PageRank scoring,
field indexing API, etc....
--
Doğacan Güney
Re: Nutchbase design doc
Posted by Doğacan Güney <do...@gmail.com>.
Hi Alex,
On Sat, Jul 3, 2010 at 14:45, Alex McLintock <al...@gmail.com>wrote:
> Doğacan
>
>
> 2010/7/3 Doğacan Güney <do...@gmail.com>:
> > I am attaching first draft of a complete nutchbase design document. There
> > are parts missing and parts not yet explained clearly but I would like to
> > get everyone's opinion on what they think so far.
>
> Thanks. I read your design and found it quite clear - at least to this
> non committer. :-)
> I would suggest that we should take this opportunity to do a full
> design document including design which has not changed from v1 to v2.
> So more please!
>
> I have made the odd comment which was then explained later on. I guess
> that meant I was a bit confused :-)
>
>
Thanks for the excellent comments. I will try to explain as best as I can.
> > Please let me
> > know which parts are unclear, which parts make no sense etc, and I will
> > improve the draft.
>
>
> The main thing I missed was any kind of overview of the data flow. I'd
> like to see a description of how a url/webpage goes through our system
> from being unknown, injected or discovered, to being queued for
> fetching in the generator, to fetched, parsed, fetchscheduled, scored,
> and generated again.
> Plus of course, indexed by sending to solr and being seen by an end
> user application.
>
> At each stage I'd like to see where the data is stored (text file,
> hbase, solr) and especially how this differs from the previous (text
> file, nutch crawldb, solr)
>
> I know that some of this may sound like a tutorial, but it is worth
> doing now rather than putting it off until later.
>
One of the things nutchbase attempts to do is hide all the complexity of
managing individual segments,
crawl/link/whatever dbs from the user. Now, nutch delegates all storage
handling to Gora (http://github.com/enis/gora).
What Gora does is, it gives you a key-value store (in this case, keys are
reversed URLs, values are WebPage objects), and
you do all your work through these objects. So storage will not be an issue
for you. Right now, Gora (and thus nutch) supports
storing your data in hbase and sql (with cassandra and other backends coming
soon).
So with nutch and gora, you will start up your hbase/sql/cassandra/etc
server(s), then nutch will figure out what to store and where.
>
> -----------------------------------------------------------------------------------------------------------------------------------------
> > Nutchbase
> > =========
> > 1) Rationale
> > * All your data in a central location (at least, nutch gives you the
> > illusion of a centralized storage)
>
> But hbase is distributed across your hadoop cluster, right? This is
> the "illusion" you meant.
>
>
Yes. Also, cassandra will be distributed too. Maybe in the future someone
will
write a HDFS-backed backend to Gora, then your data will actually live in
separate
files, but will still look like one centralized storage to you.
>
> > 2) Design
> > As mentioned above, all data for a URL is stored in a WebPage object.
> This
> > object is accessed by a key that is the reverse form of a URL. For
> example,
> > http://bar.foo.com:8983/to/index.html?a=b becomes
> > com.foo.bar:http:8983/to/index.html?a=b
>
> This was clear and is the main point to convey :-) I would in fact
> like loads more info on the WebPage object.
>
>
WebPage contains all data we have for a URL. Think Content + Parse Text +
Parse Data +
Crawl Datum + Outlinks + Inlinks...
> > If URLs are stored lexicographically, this means that URLs from same
> domain
> > and host are stored closer together. This will hopefully make developing
> > statistics tools easier for hosts and domains.
>
>
> I am unconvinced by this. Yes we want host urls together so that we
> can easily do polite fetching from individual hosts. But would it make
> statistics tools easier? Maybe i don't know enough about hbase to be
> sure.
>
>
This is not about polite fetching but let's say you want to count the number
of fetched URLs from host foo.com.
All you would have to do is to execute a scan (in hbase lingo, in gora these
are called queries), between the start
of foo.com and end of it. Since all URLs within a host are stored together,
you do not have to go over the entire
table to compute these statistics. Makes sense?
> > Writing a MapReduce job that uses Gora for storage does not take much
> > effort.
>
>
> This was confusing me. I thought that using Gora meant that we were
> losing the benefits of hdfs. So if we run a map reduce job over
> machines which are also HBase nodes does their input come from the
> hbase data stored on those nodes to reduce internal network traffic?
>
>
IIRC, we do that in Gora already. But even if we don't (which means we
forgot to do it),
using Gora means that you deal with straightforward java objects and Gora
figures out
what to store and where. As I said, your data can also be in sql, cassandra,
etc.
I guess part of the confusion is that the project was called nutchbase
(hence, implying
it is about tying nutch into hbase). But it was just a stupid name I made up
:). It is actually
about abstracting storage code away from nutch. Hope this makes it clear.
>
> > specifies all the fields that this job will be reading. If plugins will
> run
> > during a job (for example, during ParserJob, several plugins will be
> > active), then before job is run, those plugins must be initialized and
> > FieldPluggable#getFields must be called for all plugins to figure out
> which
> > fields they want to read.
>
> I'm not sure I understand how plugins are going to change in nutch version
> 2.
> Will everything need to be rewritten as the API wont look anything
> like the same?
> They will need to manipulate WebObject and not ParseData or CrawlDatum?
>
>
Yes, that's the point. Consider these two methods for scoring filters:
public void passScoreBeforeParsing(Text url, CrawlDatum datum, Content
content) throws ScoringFilterException;
public void passScoreAfterParsing(Text url, Content content, Parse parse)
throws ScoringFilterException;
Their only purpose is, as their name implies, to pass data to and from parse
phase. This is because, during parse
we do not read crawl_generate dir thus we do not have access to the crawl
datum object (hence, you read something
from crawl datum, pass it onto Content's metadata, read it there again, pass
it onto parse's metadata, etc). Instead of all
that, a plugin now simply specifies what it wants to read. Thus most plugins
just need two arguments: Reversed URL and WebPage object.
> > Even though some of these objects are still there, most of the
> CrawlDatum,
> > Content, ParseData or similar objects are removed (or are slated to be
> > removed). In most cases, plugins will simply take (String key, WebPage
> page)
> > as arguments and modify WebPage object in-place.
>
> This sort of makes sense, but worries me. It means more work for
> people to change their plugins. Can everyone think about how to make
> this as easy as possible?
>
> > 3) Jobs
> > Nutchbase uses the concept of a marker to identify what has been
> processed
> > and what will be processed from now on. WebPage object contains a
> > map<String, String> called markers. For example, when GeneratorJob
> generates
> > a URL, it puts a unique string and a unique crawl id to this marker map.
> > Then FetcherJob only fetches a given URL if it contains this marker. At
> the
> > end of the crawl cycle, DbUpdaterJob clears all markers (except markers
> > placed by IndexerJob).
>
> OK, I think I understood that but we need more detail. Can we have
> multiple crawls going on at once? Can we have parsing and other
> processing going on while there are crawls?
>
> I am assuming that almost every job has a job id and so multiple jobs
> can be run simultaneously.
>
>
Yes, furthermore, at the end of GeneratorJob, it gives you a crawl id (which
is just a string). Every URL
that has been generated at that cycle is uniquely identified by that crawl
id. So you can run two generates
(thus, have two different crawl ids), fetch them together, parse them
separate, etc...
> > a) InjectorJob:
>
> simples
>
> > b) GeneratorJob: GeneratorJob is similar to what it was before, but it is
> > now a single job. During map phase:
> > if FetchSchedule indicates given URL is to be fetched then,
> > Calculate generator score using scoring filters
> > Output <SelectorEntry<URL, score>, WebPage>
>
> I was a bit worried about this because it meant fetchschedule can't
> know anything about the score. but maybe that is because I am putting
> too much work onto the fetchschedule.
>
>
>
If a fetch schedule wants to know about score, all it has to do is to ask
for "score" field :)
Calculating generator score using scoring filters is after fetch schedule
step, because, well,
it is already like that in current nutch design :)
>
> > Then URLPartitioner partitions according to choices specified in config
> > files (by host, by ip or by domain).
>
> eh? don't understand.
>
>
Again, this is just like current nutch design. We want all URLs in a single
host (or domain, or IP
depending on your config settings) want to end up in the same reducer.
Partitioners ensure that.
> > Reduce phase counts URL according to topN and host/domain limits and
> marks
> > all URLs if limits are not yet reached.
>
> ??
>
Yeah, this part is not clear at all :)
You can tell generate to generate at most 10 URLs per host. Reduce phase
counts the number of URLs
that have been generated from that host so far, and if the limit is
exceeded, then no more URLs from
that host are generated.
>
> > GeneratorJob markes all generated URLs with a unique crawl id. Fetcher,
> > parser and indexer jobs can then use this crawl id to process only URLs
> that
> > are in that particular crawl cycle. Alternatively, they can also work on
> all
> > URLs (though, again, FetcherJob will only work on a URL which has been
> > marked by GenerateJob. So even if FetcherJob is instructed to work on
> "all"
> > URLs, it will skip those without generator markers).
>
> OK, I would like a bit more explanation of this :-)
>
>
I tried to give an explanation of crawl ids above. Did that make it clear?
Here is an example,
you have 3 URLs: http://a.com/, http://b.com/, http://c.com/. You run two
generates consecutively
(but let's say, you change URL filters between generates). In the first run,
http://a.com/ is generated
and marked with a crawl id, say, "X". Then, in the second run,
http://b.com/is generated and marked
with crawl id, "Y". Third URL is not generated in any of these runs.
Now you can tell fetcher to fetch URLs that have crawl id "X", or "Y", or
simply say "fetch all URLs
that have been marked. I don't care about crawl ids".
>
> > GeneratorJob will also print crawl id to console.
> > c) FetcherJob: During map phase, FetcherJob outputs <Integer,
> > FetcherEntry<key, WebPage>> pairs. The first member of the pair (integer)
> is
> > a random integer between 0 and 65536. After map, these pairs are
> partitioned
> > into reduces according to URL's hosts. Reduce works exactly like old
> > Fetcher's map. FetcherJob can also continue interrupted fetches now (by
> > giving "-continue" switch on command line).
>
> How do we stop fetches to interrupt them?
>
>
Ctrl+c or killing the job in JobTracker's UI. It may also be that fetcher
may fail somehow. If it does,
you do not lose your entire fetch.
> > The random integer may seem pointless but it is actually quite important
> for
> > performance. Let's say instead of a random integer we were to just output
> > <key, WebPage> pairs from map. Again, let's say you have 1000 URLs from
> host
> > a.com, and 500 URLs from b.com. Let's also say that we have 10 fetcher
> > threads so maximum queue size is 500. In this case, since all URLs from
> host
> > a.com will be processed by reduce before all URLs from b.com, during
> reduce
> > phase, only one thread will fetch URLs from a.com while every other
> thread
> > will be spin-waiting. However, with randomization, URLs from a.com and
> b.com
> > will be processed in a random order thus bandwidth utilization will be
> > higher.
>
> I'd like to see a lot more detail about this optimization. I think you
> are right, but I don't yet fully understand where the data resides in
> hadoop, and what order things gets done in.
>
>
But the point is you do not need to know that :) Data is stored as <reversed
URL, WebPage>
pairs and you get them in key-lexicographic order during MapReduce.
The last thing I say may not be completely true :) For now, all gora
backends indeed return
data to you in sorted order, but maybe other backends may not do that in the
future.
>
> > d) ParserJob: ParserJob is straightforward. It is only a map (i.e., has 0
> > reducers). Simple parses all URLs with active parse plugins.
>
> Can you explain that last sentence some more? How do we decide which
> urls need to be parsed? are they not labelled with a jobid, or crawl
> id?
>
>
ParserJob will only attempt to parse a URL if it has been marked by Fetcher.
Other than
that, the same logic applies (so you can say "parse all urls with crawl id
X, or Y, or parse all urls
that were marked by fetcher).
> > e) DbUpdaterJob: This is a combination of updatedb and invertlinks jobs.
> If
> > a URL is successfully parsed (which means, it will contain a parse
> marker),
>
> parse marker?
>
> > DbUpdaterJob will put its own marker. Note: It may make more sense to put
> a
> > marker even if a URL is not successfully parsed. DbUpdaterJob also cleans
> > all other markers.
>
> > f) IndexerJob: Goes over all URLs with a db update marker (again, you can
> > specify ALL URLs with update markers, or a crawl id), and indexes them.
>
> db update marker?
>
>
Yes, each job will put its own marker to the URL. This indicates that next
step can
process it.
> > 4) What's missing
> > Most of the core functionality and plugins have been ported.
>
> Oh cool. That worried me earlier.
>
> > However, some
> > tools and features are still missing: arc segment tools, PageRank
> scoring,
> > field indexing API, etc....
> > --
> > Doğacan Güney
> >
> >
>
> THANKS.
>
--
Doğacan Güney
Re: Nutchbase design doc
Posted by Alex McLintock <al...@gmail.com>.
Doğacan
2010/7/3 Doğacan Güney <do...@gmail.com>:
> I am attaching first draft of a complete nutchbase design document. There
> are parts missing and parts not yet explained clearly but I would like to
> get everyone's opinion on what they think so far.
Thanks. I read your design and found it quite clear - at least to this
non committer. :-)
I would suggest that we should take this opportunity to do a full
design document including design which has not changed from v1 to v2.
So more please!
I have made the odd comment which was then explained later on. I guess
that meant I was a bit confused :-)
> Please let me
> know which parts are unclear, which parts make no sense etc, and I will
> improve the draft.
The main thing I missed was any kind of overview of the data flow. I'd
like to see a description of how a url/webpage goes through our system
from being unknown, injected or discovered, to being queued for
fetching in the generator, to fetched, parsed, fetchscheduled, scored,
and generated again.
Plus of course, indexed by sending to solr and being seen by an end
user application.
At each stage I'd like to see where the data is stored (text file,
hbase, solr) and especially how this differs from the previous (text
file, nutch crawldb, solr)
I know that some of this may sound like a tutorial, but it is worth
doing now rather than putting it off until later.
-----------------------------------------------------------------------------------------------------------------------------------------
> Nutchbase
> =========
> 1) Rationale
> * All your data in a central location (at least, nutch gives you the
> illusion of a centralized storage)
But hbase is distributed across your hadoop cluster, right? This is
the "illusion" you meant.
> 2) Design
> As mentioned above, all data for a URL is stored in a WebPage object. This
> object is accessed by a key that is the reverse form of a URL. For example,
> http://bar.foo.com:8983/to/index.html?a=b becomes
> com.foo.bar:http:8983/to/index.html?a=b
This was clear and is the main point to convey :-) I would in fact
like loads more info on the WebPage object.
> If URLs are stored lexicographically, this means that URLs from same domain
> and host are stored closer together. This will hopefully make developing
> statistics tools easier for hosts and domains.
I am unconvinced by this. Yes we want host urls together so that we
can easily do polite fetching from individual hosts. But would it make
statistics tools easier? Maybe i don't know enough about hbase to be
sure.
> Writing a MapReduce job that uses Gora for storage does not take much
> effort.
This was confusing me. I thought that using Gora meant that we were
losing the benefits of hdfs. So if we run a map reduce job over
machines which are also HBase nodes does their input come from the
hbase data stored on those nodes to reduce internal network traffic?
> specifies all the fields that this job will be reading. If plugins will run
> during a job (for example, during ParserJob, several plugins will be
> active), then before job is run, those plugins must be initialized and
> FieldPluggable#getFields must be called for all plugins to figure out which
> fields they want to read.
I'm not sure I understand how plugins are going to change in nutch version 2.
Will everything need to be rewritten as the API wont look anything
like the same?
They will need to manipulate WebObject and not ParseData or CrawlDatum?
> Even though some of these objects are still there, most of the CrawlDatum,
> Content, ParseData or similar objects are removed (or are slated to be
> removed). In most cases, plugins will simply take (String key, WebPage page)
> as arguments and modify WebPage object in-place.
This sort of makes sense, but worries me. It means more work for
people to change their plugins. Can everyone think about how to make
this as easy as possible?
> 3) Jobs
> Nutchbase uses the concept of a marker to identify what has been processed
> and what will be processed from now on. WebPage object contains a
> map<String, String> called markers. For example, when GeneratorJob generates
> a URL, it puts a unique string and a unique crawl id to this marker map.
> Then FetcherJob only fetches a given URL if it contains this marker. At the
> end of the crawl cycle, DbUpdaterJob clears all markers (except markers
> placed by IndexerJob).
OK, I think I understood that but we need more detail. Can we have
multiple crawls going on at once? Can we have parsing and other
processing going on while there are crawls?
I am assuming that almost every job has a job id and so multiple jobs
can be run simultaneously.
> a) InjectorJob:
simples
> b) GeneratorJob: GeneratorJob is similar to what it was before, but it is
> now a single job. During map phase:
> if FetchSchedule indicates given URL is to be fetched then,
> Calculate generator score using scoring filters
> Output <SelectorEntry<URL, score>, WebPage>
I was a bit worried about this because it meant fetchschedule can't
know anything about the score. but maybe that is because I am putting
too much work onto the fetchschedule.
> Then URLPartitioner partitions according to choices specified in config
> files (by host, by ip or by domain).
eh? don't understand.
> Reduce phase counts URL according to topN and host/domain limits and marks
> all URLs if limits are not yet reached.
??
> GeneratorJob markes all generated URLs with a unique crawl id. Fetcher,
> parser and indexer jobs can then use this crawl id to process only URLs that
> are in that particular crawl cycle. Alternatively, they can also work on all
> URLs (though, again, FetcherJob will only work on a URL which has been
> marked by GenerateJob. So even if FetcherJob is instructed to work on "all"
> URLs, it will skip those without generator markers).
OK, I would like a bit more explanation of this :-)
> GeneratorJob will also print crawl id to console.
> c) FetcherJob: During map phase, FetcherJob outputs <Integer,
> FetcherEntry<key, WebPage>> pairs. The first member of the pair (integer) is
> a random integer between 0 and 65536. After map, these pairs are partitioned
> into reduces according to URL's hosts. Reduce works exactly like old
> Fetcher's map. FetcherJob can also continue interrupted fetches now (by
> giving "-continue" switch on command line).
How do we stop fetches to interrupt them?
> The random integer may seem pointless but it is actually quite important for
> performance. Let's say instead of a random integer we were to just output
> <key, WebPage> pairs from map. Again, let's say you have 1000 URLs from host
> a.com, and 500 URLs from b.com. Let's also say that we have 10 fetcher
> threads so maximum queue size is 500. In this case, since all URLs from host
> a.com will be processed by reduce before all URLs from b.com, during reduce
> phase, only one thread will fetch URLs from a.com while every other thread
> will be spin-waiting. However, with randomization, URLs from a.com and b.com
> will be processed in a random order thus bandwidth utilization will be
> higher.
I'd like to see a lot more detail about this optimization. I think you
are right, but I don't yet fully understand where the data resides in
hadoop, and what order things gets done in.
> d) ParserJob: ParserJob is straightforward. It is only a map (i.e., has 0
> reducers). Simple parses all URLs with active parse plugins.
Can you explain that last sentence some more? How do we decide which
urls need to be parsed? are they not labelled with a jobid, or crawl
id?
> e) DbUpdaterJob: This is a combination of updatedb and invertlinks jobs. If
> a URL is successfully parsed (which means, it will contain a parse marker),
parse marker?
> DbUpdaterJob will put its own marker. Note: It may make more sense to put a
> marker even if a URL is not successfully parsed. DbUpdaterJob also cleans
> all other markers.
> f) IndexerJob: Goes over all URLs with a db update marker (again, you can
> specify ALL URLs with update markers, or a crawl id), and indexes them.
db update marker?
> 4) What's missing
> Most of the core functionality and plugins have been ported.
Oh cool. That worried me earlier.
> However, some
> tools and features are still missing: arc segment tools, PageRank scoring,
> field indexing API, etc....
> --
> Doğacan Güney
>
>
THANKS.
Re: Nutchbase design doc
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Doğacan,
OK, cool. Comments below:
>> My only question: why introduce a new data structure called “Markers” when
>> all that seems to be is a Metadata object. Let’s use
>> o.a.tika.metadata.Metadata to represent that? My only comment then would be,
>> aren’t we still doing something you mentioned you wanted to get rid of below,
>> where you said: “For example, during parsing we don't have access to a URL's
>> fetch status. So we copy fetch status into content metadata.” Aren’t we just
>> doing the same thing with Markers?
>>
>
> Actually, markers used to be stored in the metadata object in WebPage
> (metadata is a map from string to bytes). It just seemed clearer to me to put
> it into its own field. We can discuss if moving it back into metadata makes
> more sense.
Ok, gotcha, the creation of a new data structure cleanly identified your
intention that this just wasn't "Metadata" in its vanilla sense, but
actually a first class object called Markers for the purpose of tracking
where we are in a crawl cycle. Maybe we should be even more explicit then
and call this class: "CrawlCycleTracker". I think that would make it even
more explicit.
>
> One thing: We can't use tika's metadata object as WebPage object is generated
> from an avro schema.
Not to push this, because in the end I don't feel super strongly about it,
but can't Avro handle data types such as Tika's metadata object? Or does it
only do primitive types?
>
> As for your last comment: Markers are only used to identify where we are in a
> crawl cycle and the individual crawl ids. So during parse, when we get a URL
> during MapReduce, parse can easily check if that URL has been fetched in
> *that* crawl cycle (since there is no point in parsing it if it hasn't been
> fetched). So it is not used to pass any important information around. It is
> just a simple tracking system. Did this make it any clearer?
Gotcha. In the end you could use the o.a.tika.metadata.Metadata object for
this anyways, but it wouldn't really be Metadata in its purest sense, or it
would but it's a real specific kind of Metadata: it's a (set of) marker(s)
to identify where we are in a crawl cycle.
So, +1 to having an explicit data structure to track this. I think we should
consider renaming it though as per my comment above to make it even more
explicit.
Thanks for the explanation!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Nutchbase design doc
Posted by Doğacan Güney <do...@gmail.com>.
Hi Chris,
On Sat, Jul 3, 2010 at 18:35, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Guys,
>
> This sounds awesome. Even I could understand it, which is saying something!
> :)
>
> My only question: why introduce a new data structure called “Markers” when
> all that seems to be is a Metadata object. Let’s use
> o.a.tika.metadata.Metadata to represent that? My only comment then would be,
> aren’t we still doing something you mentioned you wanted to get rid of
> below, where you said: “For example, during parsing we don't have access to
> a URL's fetch status. So we copy fetch status into content metadata.” Aren’t
> we just doing the same thing with Markers?
>
>
Actually, markers used to be stored in the metadata object in WebPage
(metadata is a map from string to bytes). It just seemed clearer to me to
put it into its own field. We can discuss if moving it back into metadata
makes more sense.
One thing: We can't use tika's metadata object as WebPage object is
generated from an avro schema.
As for your last comment: Markers are only used to identify where we are in
a crawl cycle and the individual crawl ids. So during parse, when we get a
URL during MapReduce, parse can easily check if that URL has been fetched in
*that* crawl cycle (since there is no point in parsing it if it hasn't been
fetched). So it is not used to pass any important information around. It is
just a simple tracking system. Did this make it any clearer?
> Cheers,
> Chris
>
>
>
>
>
>
> On 7/3/10 3:01 AM, "Doğacan Güney" <do...@gmail.com> wrote:
>
> Hello everyone,
>
> I am attaching first draft of a complete nutchbase design document. There
> are parts missing and parts not yet explained clearly but I would like to
> get everyone's opinion on what they think so far. Please let me
> know which parts are unclear, which parts make no sense etc, and I will
> improve the draft.
>
>
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Nutchbase
> =========
>
> 1) Rationale
>
> * All your data in a central location (at least, nutch gives you the
> illusion of a centralized storage)
> * No more segment/crawldb/linkdb merges.
> * No more "missing" data in a job. There are a lot of places where we copy
> data from one structure to another just so that it is available in a later
> job. For example, during parsing we don't have access to a URL's fetch
> status. So we copy fetch status into content metadata. This will no longer
> be necessary after nutchbase. When writing a job or a new plugin, programmer
> only needs to specify which fields she wants to read and they will be
> available to plugin / job.
> * A much simpler data model. If you want to update a small part in a single
> record, now you have to write a MR job that reads the relevant directory,
> change the single record, remove old directory and rename new directory.
> With nutchbase, you can just update that record.
>
> 2) Design
>
> As mentioned above, all data for a URL is stored in a WebPage object. This
> object is accessed by a key that is the reverse form of a URL. For example,
>
> http://bar.foo.com:8983/to/index.html?a=b becomes com.foo.bar:
> http:8983/to/index.html?a=b
>
> If URLs are stored lexicographically, this means that URLs from same domain
> and host are stored closer together. This will hopefully make developing
> statistics tools easier for hosts and domains.
>
> Writing a MapReduce job that uses Gora for storage does not take much
> effort. There is a new class called StorageUtils that has a number of static
> methods to make setting mappers/reducers/etc easier. Here is an example
> (from GeneratorJob.java):
>
> Job job = new NutchJob(getConf(), "generate: " + crawlId);
> StorageUtils.initMapperJob(job, FIELDS, SelectorEntry.class,
> WebPage.class,
> GeneratorMapper.class, URLPartitioner.class);
> StorageUtils.initReducerJob(job, GeneratorReducer.class);
>
> An important argument is the second argument to #initMapperJob. This
> specifies all the fields that this job will be reading. If plugins will run
> during a job (for example, during ParserJob, several plugins will be
> active), then before job is run, those plugins must be initialized and
> FieldPluggable#getFields must be called for all plugins to figure out which
> fields they want to read. During map or reduce phase, modifying WebPage
> object is as simple as using the built-in setters. All changes will be
> persisted.
>
> Even though some of these objects are still there, most of the CrawlDatum,
> Content, ParseData or similar objects are removed (or are slated to be
> removed). In most cases, plugins will simply take (String key, WebPage page)
> as arguments and modify WebPage object in-place.
>
> 3) Jobs
>
> Nutchbase uses the concept of a marker to identify what has been processed
> and what will be processed from now on. WebPage object contains a
> map<String, String> called markers. For example, when GeneratorJob generates
> a URL, it puts a unique string and a unique crawl id to this marker map.
> Then FetcherJob only fetches a given URL if it contains this marker. At the
> end of the crawl cycle, DbUpdaterJob clears all markers (except markers
> placed by IndexerJob).
>
> a) InjectorJob: This phase consists of two different jobs. First job reads
> from a text file (as before) then puts a special inject marker. Second job
> goes over all URLs then if it finds a WebPage with inject marker but nothing
> else (so, it is a new URL), then this URL is injected (and marker is
> deleted). Otherwise, marker is just deleted (since URL is already injected).
>
> b) GeneratorJob: GeneratorJob is similar to what it was before, but it is
> now a single job. During map phase:
>
> if FetchSchedule indicates given URL is to be fetched then,
> Calculate generator score using scoring filters
> Output <SelectorEntry<URL, score>, WebPage>
>
> SelectorEntry is sorted according to given score so highest scoring entries
> will be processed first during reduce.
>
> Then URLPartitioner partitions according to choices specified in config
> files (by host, by ip or by domain).
>
> Reduce phase counts URL according to topN and host/domain limits and marks
> all URLs if limits are not yet reached.
>
> GeneratorJob markes all generated URLs with a unique crawl id. Fetcher,
> parser and indexer jobs can then use this crawl id to process only URLs that
> are in that particular crawl cycle. Alternatively, they can also work on all
> URLs (though, again, FetcherJob will only work on a URL which has been
> marked by GenerateJob. So even if FetcherJob is instructed to work on "all"
> URLs, it will skip those without generator markers).
>
> GeneratorJob will also print crawl id to console.
>
> c) FetcherJob: During map phase, FetcherJob outputs <Integer,
> FetcherEntry<key, WebPage>> pairs. The first member of the pair (integer) is
> a random integer between 0 and 65536. After map, these pairs are partitioned
> into reduces according to URL's hosts. Reduce works exactly like old
> Fetcher's map. FetcherJob can also continue interrupted fetches now (by
> giving "-continue" switch on command line).
>
> The random integer may seem pointless but it is actually quite important
> for performance. Let's say instead of a random integer we were to just
> output <key, WebPage> pairs from map. Again, let's say you have 1000 URLs
> from host a.com <http://a.com> , and 500 URLs from b.com <http://b.com> .
> Let's also say that we have 10 fetcher threads so maximum queue size is 500.
> In this case, since all URLs from host a.com <http://a.com> will be
> processed by reduce before all URLs from b.com <http://b.com> , during
> reduce phase, only one thread will fetch URLs from a.com <http://a.com>
> while every other thread will be spin-waiting. However, with randomization,
> URLs from a.com <http://a.com> and b.com <http://b.com> will be
> processed in a random order thus bandwidth utilization will be higher.
>
>
> d) ParserJob: ParserJob is straightforward. It is only a map (i.e., has 0
> reducers). Simple parses all URLs with active parse plugins.
>
> e) DbUpdaterJob: This is a combination of updatedb and invertlinks jobs. If
> a URL is successfully parsed (which means, it will contain a parse marker),
> DbUpdaterJob will put its own marker. Note: It may make more sense to put a
> marker even if a URL is not successfully parsed. DbUpdaterJob also cleans
> all other markers.
>
> f) IndexerJob: Goes over all URLs with a db update marker (again, you can
> specify ALL URLs with update markers, or a crawl id), and indexes them.
>
> 4) What's missing
>
> Most of the core functionality and plugins have been ported. However, some
> tools and features are still missing: arc segment tools, PageRank scoring,
> field indexing API, etc....
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: *Chris.Mattmann@jpl.nasa.gov
> *WWW: *http://sunset.usc.edu/~mattmann/
> *++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
--
Doğacan Güney
Re: Nutchbase design doc
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Guys,
This sounds awesome. Even I could understand it, which is saying something! :)
My only question: why introduce a new data structure called “Markers” when all that seems to be is a Metadata object. Let’s use o.a.tika.metadata.Metadata to represent that? My only comment then would be, aren’t we still doing something you mentioned you wanted to get rid of below, where you said: “For example, during parsing we don't have access to a URL's fetch status. So we copy fetch status into content metadata.” Aren’t we just doing the same thing with Markers?
Cheers,
Chris
On 7/3/10 3:01 AM, "Doğacan Güney" <do...@gmail.com> wrote:
Hello everyone,
I am attaching first draft of a complete nutchbase design document. There are parts missing and parts not yet explained clearly but I would like to get everyone's opinion on what they think so far. Please let me
know which parts are unclear, which parts make no sense etc, and I will improve the draft.
-----------------------------------------------------------------------------------------------------------------------------------------
Nutchbase
=========
1) Rationale
* All your data in a central location (at least, nutch gives you the illusion of a centralized storage)
* No more segment/crawldb/linkdb merges.
* No more "missing" data in a job. There are a lot of places where we copy data from one structure to another just so that it is available in a later job. For example, during parsing we don't have access to a URL's fetch status. So we copy fetch status into content metadata. This will no longer be necessary after nutchbase. When writing a job or a new plugin, programmer only needs to specify which fields she wants to read and they will be available to plugin / job.
* A much simpler data model. If you want to update a small part in a single record, now you have to write a MR job that reads the relevant directory, change the single record, remove old directory and rename new directory. With nutchbase, you can just update that record.
2) Design
As mentioned above, all data for a URL is stored in a WebPage object. This object is accessed by a key that is the reverse form of a URL. For example,
http://bar.foo.com:8983/to/index.html?a=b becomes com.foo.bar:http:8983/to/index.html?a=b
If URLs are stored lexicographically, this means that URLs from same domain and host are stored closer together. This will hopefully make developing statistics tools easier for hosts and domains.
Writing a MapReduce job that uses Gora for storage does not take much effort. There is a new class called StorageUtils that has a number of static methods to make setting mappers/reducers/etc easier. Here is an example (from GeneratorJob.java):
Job job = new NutchJob(getConf(), "generate: " + crawlId);
StorageUtils.initMapperJob(job, FIELDS, SelectorEntry.class, WebPage.class,
GeneratorMapper.class, URLPartitioner.class);
StorageUtils.initReducerJob(job, GeneratorReducer.class);
An important argument is the second argument to #initMapperJob. This specifies all the fields that this job will be reading. If plugins will run during a job (for example, during ParserJob, several plugins will be active), then before job is run, those plugins must be initialized and FieldPluggable#getFields must be called for all plugins to figure out which fields they want to read. During map or reduce phase, modifying WebPage object is as simple as using the built-in setters. All changes will be persisted.
Even though some of these objects are still there, most of the CrawlDatum, Content, ParseData or similar objects are removed (or are slated to be removed). In most cases, plugins will simply take (String key, WebPage page) as arguments and modify WebPage object in-place.
3) Jobs
Nutchbase uses the concept of a marker to identify what has been processed and what will be processed from now on. WebPage object contains a map<String, String> called markers. For example, when GeneratorJob generates a URL, it puts a unique string and a unique crawl id to this marker map. Then FetcherJob only fetches a given URL if it contains this marker. At the end of the crawl cycle, DbUpdaterJob clears all markers (except markers placed by IndexerJob).
a) InjectorJob: This phase consists of two different jobs. First job reads from a text file (as before) then puts a special inject marker. Second job goes over all URLs then if it finds a WebPage with inject marker but nothing else (so, it is a new URL), then this URL is injected (and marker is deleted). Otherwise, marker is just deleted (since URL is already injected).
b) GeneratorJob: GeneratorJob is similar to what it was before, but it is now a single job. During map phase:
if FetchSchedule indicates given URL is to be fetched then,
Calculate generator score using scoring filters
Output <SelectorEntry<URL, score>, WebPage>
SelectorEntry is sorted according to given score so highest scoring entries will be processed first during reduce.
Then URLPartitioner partitions according to choices specified in config files (by host, by ip or by domain).
Reduce phase counts URL according to topN and host/domain limits and marks all URLs if limits are not yet reached.
GeneratorJob markes all generated URLs with a unique crawl id. Fetcher, parser and indexer jobs can then use this crawl id to process only URLs that are in that particular crawl cycle. Alternatively, they can also work on all URLs (though, again, FetcherJob will only work on a URL which has been marked by GenerateJob. So even if FetcherJob is instructed to work on "all" URLs, it will skip those without generator markers).
GeneratorJob will also print crawl id to console.
c) FetcherJob: During map phase, FetcherJob outputs <Integer, FetcherEntry<key, WebPage>> pairs. The first member of the pair (integer) is a random integer between 0 and 65536. After map, these pairs are partitioned into reduces according to URL's hosts. Reduce works exactly like old Fetcher's map. FetcherJob can also continue interrupted fetches now (by giving "-continue" switch on command line).
The random integer may seem pointless but it is actually quite important for performance. Let's say instead of a random integer we were to just output <key, WebPage> pairs from map. Again, let's say you have 1000 URLs from host a.com <http://a.com> , and 500 URLs from b.com <http://b.com> . Let's also say that we have 10 fetcher threads so maximum queue size is 500. In this case, since all URLs from host a.com <http://a.com> will be processed by reduce before all URLs from b.com <http://b.com> , during reduce phase, only one thread will fetch URLs from a.com <http://a.com> while every other thread will be spin-waiting. However, with randomization, URLs from a.com <http://a.com> and b.com <http://b.com> will be processed in a random order thus bandwidth utilization will be higher.
d) ParserJob: ParserJob is straightforward. It is only a map (i.e., has 0 reducers). Simple parses all URLs with active parse plugins.
e) DbUpdaterJob: This is a combination of updatedb and invertlinks jobs. If a URL is successfully parsed (which means, it will contain a parse marker), DbUpdaterJob will put its own marker. Note: It may make more sense to put a marker even if a URL is not successfully parsed. DbUpdaterJob also cleans all other markers.
f) IndexerJob: Goes over all URLs with a db update marker (again, you can specify ALL URLs with update markers, or a crawl id), and indexes them.
4) What's missing
Most of the core functionality and plugins have been ported. However, some tools and features are still missing: arc segment tools, PageRank scoring, field indexing API, etc....
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++