You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Meryl Silverburgh <si...@gmail.com> on 2007/04/16 05:32:38 UTC
Crawl www.yahoo.com with nutch
I setup nutch to crawl, in my input file, I only have 1 site
"http://www.yahoo.com"
$ bin/nutch crawl urls -dir crawl -depth 3
and I have added 'yahoo.com' as my domain name in crawl-urlfilter.txt
# accept hosts in MY.DOMAIN.NAME
+^http://([a-zA-Z0-9]*\.)*(cnn.com|yahoo.com)/
But no links is being fetched, when I change the link to www.cnn.com,
it works. Can you please tell me what do I need to work to make
www.yahoo.com works?
$ bin/nutch crawl urls -dir crawl -depth 3
crawl started in: crawl
rootUrlDir = urls
threads = 10
depth = 3
Injector: starting
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: crawl/segments/20070415222440
Generator: filtering: false
Generator: topN: 2147483647
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls by host, for politeness.
Generator: done.
Fetcher: starting
Fetcher: segment: crawl/segments/20070415222440
Fetcher: threads: 10
fetching http://www.yahoo.com/
Fetcher: done
CrawlDb update: starting
CrawlDb update: db: crawl/crawldb
CrawlDb update: segments: [crawl/segments/20070415222440]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: crawl/segments/20070415222449
Generator: filtering: false
Generator: topN: 2147483647
Generator: jobtracker is 'local', generating exactly one partition.
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=1 - no more URLs to fetch.
LinkDb: starting
LinkDb: linkdb: crawl/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: adding segment: crawl/segments/20070415222440
LinkDb: done
Indexer: starting
Indexer: linkdb: crawl/linkdb
Indexer: adding segment: crawl/segments/20070415222440
Indexing [http://www.yahoo.com/] with analyzer
org.apache.nutch.analysis.NutchDocumentAnalyzer@76ab2f (null)
Optimizing index.
merging segments _ram_0 (1 docs) into _0 (1 docs)
Indexer: done
Dedup: starting
Dedup: adding indexes in: crawl/indexes
Dedup: done
merging indexes to: crawl/index
Adding crawl/indexes/part-00000
done merging
crawl finished: crawl
Re: Re: Crawl www.yahoo.com with nutch
Posted by songjue <so...@gmail.com>.
Well, I met that before. Make sure you have set the "http.agent.name" property in conf/nutch-default.xml
songjue
2007-04-16
发件人： Meryl Silverburgh
发送时间： 2007-04-16 12:08:26
收件人： nutch-user@lucene.apache.org
抄送：
主题： Re: Crawl www.yahoo.com with nutch
I am using 0.9 too.
I am now getting further, but I get a bunch of NullPointerException:
fetch of http://www.yahoo.com/s/557760 failed with:
java.lang.NullPointerException
fetch of http://www.yahoo.com/r/hq failed with: java.lang.NullPointerException
fetch of http://www.yahoo.com/s/557762 failed with:
java.lang.NullPointerException
On 4/15/07, songjue <songjue@gmail.com > wrote:
> I try this, Nutch0.9 works just fine. What's your Nutch version?
>
>
>
> songjue
> 2007-04-16
>
>
>
> 发件人： Meryl Silverburgh
> 发送时间： 2007-04-16 11:33:05
> 收件人： nutch-user@lucene.apache.org
> 抄送：
> 主题： Crawl www.yahoo.com with nutch
>
> I setup nutch to crawl, in my input file, I only have 1 site
> "http://www.yahoo.com"
>
> $ bin/nutch crawl urls -dir crawl -depth 3
>
> and I have added 'yahoo.com' as my domain name in crawl-urlfilter.txt
>
> # accept hosts in MY.DOMAIN.NAME
> +^http://([a-zA-Z0-9]*\.)*(cnn.com|yahoo.com)/
>
> But no links is being fetched, when I change the link to www.cnn.com,
> it works. Can you please tell me what do I need to work to make
> www.yahoo.com works?
>
> $ bin/nutch crawl urls -dir crawl -depth 3
> crawl started in: crawl
> rootUrlDir = urls
> threads = 10
> depth = 3
> Injector: starting
> Injector: crawlDb: crawl/crawldb
> Injector: urlDir: urls
> Injector: Converting injected urls to crawl db entries.
> Injector: Merging injected urls into crawl db.
> Injector: done
> Generator: Selecting best-scoring urls due for fetch.
> Generator: starting
> Generator: segment: crawl/segments/20070415222440
> Generator: filtering: false
> Generator: topN: 2147483647
> Generator: jobtracker is 'local', generating exactly one partition.
> Generator: Partitioning selected urls by host, for politeness.
> Generator: done.
> Fetcher: starting
> Fetcher: segment: crawl/segments/20070415222440
> Fetcher: threads: 10
> fetching http://www.yahoo.com/
> Fetcher: done
> CrawlDb update: starting
> CrawlDb update: db: crawl/crawldb
> CrawlDb update: segments: [crawl/segments/20070415222440]
> CrawlDb update: additions allowed: true
> CrawlDb update: URL normalizing: true
> CrawlDb update: URL filtering: true
> CrawlDb update: Merging segment data into db.
> CrawlDb update: done
> Generator: Selecting best-scoring urls due for fetch.
> Generator: starting
> Generator: segment: crawl/segments/20070415222449
> Generator: filtering: false
> Generator: topN: 2147483647
> Generator: jobtracker is 'local', generating exactly one partition.
> Generator: 0 records selected for fetching, exiting ...
> Stopping at depth=1 - no more URLs to fetch.
> LinkDb: starting
> LinkDb: linkdb: crawl/linkdb
> LinkDb: URL normalize: true
> LinkDb: URL filter: true
> LinkDb: adding segment: crawl/segments/20070415222440
> LinkDb: done
> Indexer: starting
> Indexer: linkdb: crawl/linkdb
> Indexer: adding segment: crawl/segments/20070415222440
> Indexing [http://www.yahoo.com/] with analyzer
> org.apache.nutch.analysis.NutchDocumentAnalyzer@76ab2f (null)
> Optimizing index.
> merging segments _ram_0 (1 docs) into _0 (1 docs)
> Indexer: done
> Dedup: starting
> Dedup: adding indexes in: crawl/indexes
> Dedup: done
> merging indexes to: crawl/index
> Adding crawl/indexes/part-00000
> done merging
> crawl finished: crawl
>
Re: Re: Re: Crawl www.yahoo.com with nutch
Posted by songjue <so...@gmail.com>.
It's a cool tool, you can browse and manage index you crawl down, e.g.,
it can display number of documents, fields, terms in your index, do search,
and much more. Wish it be helpful for you.
songjue
2007-04-17
发件人： Meryl Silverburgh
发送时间： 2007-04-17 02:35:34
收件人： nutch-user@lucene.apache.org
抄送：
主题： Re: Re: Crawl www.yahoo.com with nutch
no. I am not. Can you please tell me how can I use Luke to troble
shoot my problem?
On 4/16/07, songjue <songjue@gmail.com > wrote:
> Really strang. Did you try Luke? Its's much more convenient for debugging.
>
>
>
>
> songjue
> 2007-04-16
>
>
>
> 发件人： Meryl Silverburgh
> 发送时间： 2007-04-16 12:15:39
> 收件人： nutch-user@lucene.apache.org
> 抄送：
> 主题： Re: Crawl www.yahoo.com with nutch
>
> I have use this command to crawl, up to the depth of 6
> $ bin/nutch crawl urls -dir crawl -depth 6
> and then this commad to read the link
> $ bin/nutch readdb crawl/crawldb -topN 50 test
>
> but I only 10 links, can you please tell me why?
> $ more test/part-00000
> 2.1111112 http://www.yahoo.com/
> 0.11111111 http://srd.yahoo.com/hp5-v
> 0.11111111 http://www.yahoo.com/+document.cookie+
> 0.11111111 http://www.yahoo.com/1.0
> 0.11111111 http://www.yahoo.com/2.0.0
> 0.11111111 http://www.yahoo.com/r/hf
> 0.11111111 http://www.yahoo.com/r/hq
> 0.11111111 http://www.yahoo.com/r\/1m
> 0.11111111 http://www.yahoo.com/s/557762
> 0.11111111 http://www.yahoo.com/s/557770
>
>
> On 4/15/07, Meryl Silverburgh <silverburgh.meryl@gmail.com > wrote:
> > I am using 0.9 too.
> >
> > I am now getting further, but I get a bunch of NullPointerException:
> >
> > fetch of http://www.yahoo.com/s/557760 failed with:
> > java.lang.NullPointerException
> > fetch of http://www.yahoo.com/r/hq failed with: java.lang.NullPointerException
> > fetch of http://www.yahoo.com/s/557762 failed with:
> > java.lang.NullPointerException
> >
> >
> > On 4/15/07, songjue <songjue@gmail.com > wrote:
> > > I try this, Nutch0.9 works just fine. What's your Nutch version?
> > >
> > >
> > >
> > > songjue
> > > 2007-04-16
> > >
> > >
> > >
> > > 发件人： Meryl Silverburgh
> > > 发送时间： 2007-04-16 11:33:05
> > > 收件人： nutch-user@lucene.apache.org
> > > 抄送：
> > > 主题： Crawl www.yahoo.com with nutch
> > >
> > > I setup nutch to crawl, in my input file, I only have 1 site
> > > "http://www.yahoo.com"
> > >
> > > $ bin/nutch crawl urls -dir crawl -depth 3
> > >
> > > and I have added 'yahoo.com' as my domain name in crawl-urlfilter.txt
> > >
> > > # accept hosts in MY.DOMAIN.NAME
> > > +^http://([a-zA-Z0-9]*\.)*(cnn.com|yahoo.com)/
> > >
> > > But no links is being fetched, when I change the link to www.cnn.com,
> > > it works. Can you please tell me what do I need to work to make
> > > www.yahoo.com works?
> > >
> > > $ bin/nutch crawl urls -dir crawl -depth 3
> > > crawl started in: crawl
> > > rootUrlDir = urls
> > > threads = 10
> > > depth = 3
> > > Injector: starting
> > > Injector: crawlDb: crawl/crawldb
> > > Injector: urlDir: urls
> > > Injector: Converting injected urls to crawl db entries.
> > > Injector: Merging injected urls into crawl db.
> > > Injector: done
> > > Generator: Selecting best-scoring urls due for fetch.
> > > Generator: starting
> > > Generator: segment: crawl/segments/20070415222440
> > > Generator: filtering: false
> > > Generator: topN: 2147483647
> > > Generator: jobtracker is 'local', generating exactly one partition.
> > > Generator: Partitioning selected urls by host, for politeness.
> > > Generator: done.
> > > Fetcher: starting
> > > Fetcher: segment: crawl/segments/20070415222440
> > > Fetcher: threads: 10
> > > fetching http://www.yahoo.com/
> > > Fetcher: done
> > > CrawlDb update: starting
> > > CrawlDb update: db: crawl/crawldb
> > > CrawlDb update: segments: [crawl/segments/20070415222440]
> > > CrawlDb update: additions allowed: true
> > > CrawlDb update: URL normalizing: true
> > > CrawlDb update: URL filtering: true
> > > CrawlDb update: Merging segment data into db.
> > > CrawlDb update: done
> > > Generator: Selecting best-scoring urls due for fetch.
> > > Generator: starting
> > > Generator: segment: crawl/segments/20070415222449
> > > Generator: filtering: false
> > > Generator: topN: 2147483647
> > > Generator: jobtracker is 'local', generating exactly one partition.
> > > Generator: 0 records selected for fetching, exiting ...
> > > Stopping at depth=1 - no more URLs to fetch.
> > > LinkDb: starting
> > > LinkDb: linkdb: crawl/linkdb
> > > LinkDb: URL normalize: true
> > > LinkDb: URL filter: true
> > > LinkDb: adding segment: crawl/segments/20070415222440
> > > LinkDb: done
> > > Indexer: starting
> > > Indexer: linkdb: crawl/linkdb
> > > Indexer: adding segment: crawl/segments/20070415222440
> > > Indexing [http://www.yahoo.com/] with analyzer
> > > org.apache.nutch.analysis.NutchDocumentAnalyzer@76ab2f (null)
> > > Optimizing index.
> > > merging segments _ram_0 (1 docs) into _0 (1 docs)
> > > Indexer: done
> > > Dedup: starting
> > > Dedup: adding indexes in: crawl/indexes
> > > Dedup: done
> > > merging indexes to: crawl/index
> > > Adding crawl/indexes/part-00000
> > > done merging
> > > crawl finished: crawl
> > >
> >
>
Re: Re: Crawl www.yahoo.com with nutch
Posted by Meryl Silverburgh <si...@gmail.com>.
no. I am not. Can you please tell me how can I use Luke to troble
shoot my problem?
On 4/16/07, songjue <so...@gmail.com> wrote:
> Really strang. Did you try Luke? Its's much more convenient for debugging.
>
>
>
>
> songjue
> 2007-04-16
>
>
>
> 发件人： Meryl Silverburgh
> 发送时间： 2007-04-16 12:15:39
> 收件人： nutch-user@lucene.apache.org
> 抄送：
> 主题： Re: Crawl www.yahoo.com with nutch
>
> I have use this command to crawl, up to the depth of 6
> $ bin/nutch crawl urls -dir crawl -depth 6
> and then this commad to read the link
> $ bin/nutch readdb crawl/crawldb -topN 50 test
>
> but I only 10 links, can you please tell me why?
> $ more test/part-00000
> 2.1111112 http://www.yahoo.com/
> 0.11111111 http://srd.yahoo.com/hp5-v
> 0.11111111 http://www.yahoo.com/+document.cookie+
> 0.11111111 http://www.yahoo.com/1.0
> 0.11111111 http://www.yahoo.com/2.0.0
> 0.11111111 http://www.yahoo.com/r/hf
> 0.11111111 http://www.yahoo.com/r/hq
> 0.11111111 http://www.yahoo.com/r\/1m
> 0.11111111 http://www.yahoo.com/s/557762
> 0.11111111 http://www.yahoo.com/s/557770
>
>
> On 4/15/07, Meryl Silverburgh <silverburgh.meryl@gmail.com > wrote:
> > I am using 0.9 too.
> >
> > I am now getting further, but I get a bunch of NullPointerException:
> >
> > fetch of http://www.yahoo.com/s/557760 failed with:
> > java.lang.NullPointerException
> > fetch of http://www.yahoo.com/r/hq failed with: java.lang.NullPointerException
> > fetch of http://www.yahoo.com/s/557762 failed with:
> > java.lang.NullPointerException
> >
> >
> > On 4/15/07, songjue <songjue@gmail.com > wrote:
> > > I try this, Nutch0.9 works just fine. What's your Nutch version?
> > >
> > >
> > >
> > > songjue
> > > 2007-04-16
> > >
> > >
> > >
> > > 发件人： Meryl Silverburgh
> > > 发送时间： 2007-04-16 11:33:05
> > > 收件人： nutch-user@lucene.apache.org
> > > 抄送：
> > > 主题： Crawl www.yahoo.com with nutch
> > >
> > > I setup nutch to crawl, in my input file, I only have 1 site
> > > "http://www.yahoo.com"
> > >
> > > $ bin/nutch crawl urls -dir crawl -depth 3
> > >
> > > and I have added 'yahoo.com' as my domain name in crawl-urlfilter.txt
> > >
> > > # accept hosts in MY.DOMAIN.NAME
> > > +^http://([a-zA-Z0-9]*\.)*(cnn.com|yahoo.com)/
> > >
> > > But no links is being fetched, when I change the link to www.cnn.com,
> > > it works. Can you please tell me what do I need to work to make
> > > www.yahoo.com works?
> > >
> > > $ bin/nutch crawl urls -dir crawl -depth 3
> > > crawl started in: crawl
> > > rootUrlDir = urls
> > > threads = 10
> > > depth = 3
> > > Injector: starting
> > > Injector: crawlDb: crawl/crawldb
> > > Injector: urlDir: urls
> > > Injector: Converting injected urls to crawl db entries.
> > > Injector: Merging injected urls into crawl db.
> > > Injector: done
> > > Generator: Selecting best-scoring urls due for fetch.
> > > Generator: starting
> > > Generator: segment: crawl/segments/20070415222440
> > > Generator: filtering: false
> > > Generator: topN: 2147483647
> > > Generator: jobtracker is 'local', generating exactly one partition.
> > > Generator: Partitioning selected urls by host, for politeness.
> > > Generator: done.
> > > Fetcher: starting
> > > Fetcher: segment: crawl/segments/20070415222440
> > > Fetcher: threads: 10
> > > fetching http://www.yahoo.com/
> > > Fetcher: done
> > > CrawlDb update: starting
> > > CrawlDb update: db: crawl/crawldb
> > > CrawlDb update: segments: [crawl/segments/20070415222440]
> > > CrawlDb update: additions allowed: true
> > > CrawlDb update: URL normalizing: true
> > > CrawlDb update: URL filtering: true
> > > CrawlDb update: Merging segment data into db.
> > > CrawlDb update: done
> > > Generator: Selecting best-scoring urls due for fetch.
> > > Generator: starting
> > > Generator: segment: crawl/segments/20070415222449
> > > Generator: filtering: false
> > > Generator: topN: 2147483647
> > > Generator: jobtracker is 'local', generating exactly one partition.
> > > Generator: 0 records selected for fetching, exiting ...
> > > Stopping at depth=1 - no more URLs to fetch.
> > > LinkDb: starting
> > > LinkDb: linkdb: crawl/linkdb
> > > LinkDb: URL normalize: true
> > > LinkDb: URL filter: true
> > > LinkDb: adding segment: crawl/segments/20070415222440
> > > LinkDb: done
> > > Indexer: starting
> > > Indexer: linkdb: crawl/linkdb
> > > Indexer: adding segment: crawl/segments/20070415222440
> > > Indexing [http://www.yahoo.com/] with analyzer
> > > org.apache.nutch.analysis.NutchDocumentAnalyzer@76ab2f (null)
> > > Optimizing index.
> > > merging segments _ram_0 (1 docs) into _0 (1 docs)
> > > Indexer: done
> > > Dedup: starting
> > > Dedup: adding indexes in: crawl/indexes
> > > Dedup: done
> > > merging indexes to: crawl/index
> > > Adding crawl/indexes/part-00000
> > > done merging
> > > crawl finished: crawl
> > >
> >
>
Re: Re: Crawl www.yahoo.com with nutch
Posted by songjue <so...@gmail.com>.
Really strang. Did you try Luke? Its's much more convenient for debugging.
songjue
2007-04-16
发件人： Meryl Silverburgh
发送时间： 2007-04-16 12:15:39
收件人： nutch-user@lucene.apache.org
抄送：
主题： Re: Crawl www.yahoo.com with nutch
I have use this command to crawl, up to the depth of 6
$ bin/nutch crawl urls -dir crawl -depth 6
and then this commad to read the link
$ bin/nutch readdb crawl/crawldb -topN 50 test
but I only 10 links, can you please tell me why?
$ more test/part-00000
2.1111112 http://www.yahoo.com/
0.11111111 http://srd.yahoo.com/hp5-v
0.11111111 http://www.yahoo.com/+document.cookie+
0.11111111 http://www.yahoo.com/1.0
0.11111111 http://www.yahoo.com/2.0.0
0.11111111 http://www.yahoo.com/r/hf
0.11111111 http://www.yahoo.com/r/hq
0.11111111 http://www.yahoo.com/r\/1m
0.11111111 http://www.yahoo.com/s/557762
0.11111111 http://www.yahoo.com/s/557770
On 4/15/07, Meryl Silverburgh <silverburgh.meryl@gmail.com > wrote:
> I am using 0.9 too.
>
> I am now getting further, but I get a bunch of NullPointerException:
>
> fetch of http://www.yahoo.com/s/557760 failed with:
> java.lang.NullPointerException
> fetch of http://www.yahoo.com/r/hq failed with: java.lang.NullPointerException
> fetch of http://www.yahoo.com/s/557762 failed with:
> java.lang.NullPointerException
>
>
> On 4/15/07, songjue <songjue@gmail.com > wrote:
> > I try this, Nutch0.9 works just fine. What's your Nutch version?
> >
> >
> >
> > songjue
> > 2007-04-16
> >
> >
> >
> > 发件人： Meryl Silverburgh
> > 发送时间： 2007-04-16 11:33:05
> > 收件人： nutch-user@lucene.apache.org
> > 抄送：
> > 主题： Crawl www.yahoo.com with nutch
> >
> > I setup nutch to crawl, in my input file, I only have 1 site
> > "http://www.yahoo.com"
> >
> > $ bin/nutch crawl urls -dir crawl -depth 3
> >
> > and I have added 'yahoo.com' as my domain name in crawl-urlfilter.txt
> >
> > # accept hosts in MY.DOMAIN.NAME
> > +^http://([a-zA-Z0-9]*\.)*(cnn.com|yahoo.com)/
> >
> > But no links is being fetched, when I change the link to www.cnn.com,
> > it works. Can you please tell me what do I need to work to make
> > www.yahoo.com works?
> >
> > $ bin/nutch crawl urls -dir crawl -depth 3
> > crawl started in: crawl
> > rootUrlDir = urls
> > threads = 10
> > depth = 3
> > Injector: starting
> > Injector: crawlDb: crawl/crawldb
> > Injector: urlDir: urls
> > Injector: Converting injected urls to crawl db entries.
> > Injector: Merging injected urls into crawl db.
> > Injector: done
> > Generator: Selecting best-scoring urls due for fetch.
> > Generator: starting
> > Generator: segment: crawl/segments/20070415222440
> > Generator: filtering: false
> > Generator: topN: 2147483647
> > Generator: jobtracker is 'local', generating exactly one partition.
> > Generator: Partitioning selected urls by host, for politeness.
> > Generator: done.
> > Fetcher: starting
> > Fetcher: segment: crawl/segments/20070415222440
> > Fetcher: threads: 10
> > fetching http://www.yahoo.com/
> > Fetcher: done
> > CrawlDb update: starting
> > CrawlDb update: db: crawl/crawldb
> > CrawlDb update: segments: [crawl/segments/20070415222440]
> > CrawlDb update: additions allowed: true
> > CrawlDb update: URL normalizing: true
> > CrawlDb update: URL filtering: true
> > CrawlDb update: Merging segment data into db.
> > CrawlDb update: done
> > Generator: Selecting best-scoring urls due for fetch.
> > Generator: starting
> > Generator: segment: crawl/segments/20070415222449
> > Generator: filtering: false
> > Generator: topN: 2147483647
> > Generator: jobtracker is 'local', generating exactly one partition.
> > Generator: 0 records selected for fetching, exiting ...
> > Stopping at depth=1 - no more URLs to fetch.
> > LinkDb: starting
> > LinkDb: linkdb: crawl/linkdb
> > LinkDb: URL normalize: true
> > LinkDb: URL filter: true
> > LinkDb: adding segment: crawl/segments/20070415222440
> > LinkDb: done
> > Indexer: starting
> > Indexer: linkdb: crawl/linkdb
> > Indexer: adding segment: crawl/segments/20070415222440
> > Indexing [http://www.yahoo.com/] with analyzer
> > org.apache.nutch.analysis.NutchDocumentAnalyzer@76ab2f (null)
> > Optimizing index.
> > merging segments _ram_0 (1 docs) into _0 (1 docs)
> > Indexer: done
> > Dedup: starting
> > Dedup: adding indexes in: crawl/indexes
> > Dedup: done
> > merging indexes to: crawl/index
> > Adding crawl/indexes/part-00000
> > done merging
> > crawl finished: crawl
> >
>
Re: Crawl www.yahoo.com with nutch
Posted by Meryl Silverburgh <si...@gmail.com>.
I have use this command to crawl, up to the depth of 6
$ bin/nutch crawl urls -dir crawl -depth 6
and then this commad to read the link
$ bin/nutch readdb crawl/crawldb -topN 50 test
but I only 10 links, can you please tell me why?
$ more test/part-00000
2.1111112 http://www.yahoo.com/
0.11111111 http://srd.yahoo.com/hp5-v
0.11111111 http://www.yahoo.com/+document.cookie+
0.11111111 http://www.yahoo.com/1.0
0.11111111 http://www.yahoo.com/2.0.0
0.11111111 http://www.yahoo.com/r/hf
0.11111111 http://www.yahoo.com/r/hq
0.11111111 http://www.yahoo.com/r\/1m
0.11111111 http://www.yahoo.com/s/557762
0.11111111 http://www.yahoo.com/s/557770
On 4/15/07, Meryl Silverburgh <si...@gmail.com> wrote:
> I am using 0.9 too.
>
> I am now getting further, but I get a bunch of NullPointerException:
>
> fetch of http://www.yahoo.com/s/557760 failed with:
> java.lang.NullPointerException
> fetch of http://www.yahoo.com/r/hq failed with: java.lang.NullPointerException
> fetch of http://www.yahoo.com/s/557762 failed with:
> java.lang.NullPointerException
>
>
> On 4/15/07, songjue <so...@gmail.com> wrote:
> > I try this, Nutch0.9 works just fine. What's your Nutch version?
> >
> >
> >
> > songjue
> > 2007-04-16
> >
> >
> >
> > 发件人： Meryl Silverburgh
> > 发送时间： 2007-04-16 11:33:05
> > 收件人： nutch-user@lucene.apache.org
> > 抄送：
> > 主题： Crawl www.yahoo.com with nutch
> >
> > I setup nutch to crawl, in my input file, I only have 1 site
> > "http://www.yahoo.com"
> >
> > $ bin/nutch crawl urls -dir crawl -depth 3
> >
> > and I have added 'yahoo.com' as my domain name in crawl-urlfilter.txt
> >
> > # accept hosts in MY.DOMAIN.NAME
> > +^http://([a-zA-Z0-9]*\.)*(cnn.com|yahoo.com)/
> >
> > But no links is being fetched, when I change the link to www.cnn.com,
> > it works. Can you please tell me what do I need to work to make
> > www.yahoo.com works?
> >
> > $ bin/nutch crawl urls -dir crawl -depth 3
> > crawl started in: crawl
> > rootUrlDir = urls
> > threads = 10
> > depth = 3
> > Injector: starting
> > Injector: crawlDb: crawl/crawldb
> > Injector: urlDir: urls
> > Injector: Converting injected urls to crawl db entries.
> > Injector: Merging injected urls into crawl db.
> > Injector: done
> > Generator: Selecting best-scoring urls due for fetch.
> > Generator: starting
> > Generator: segment: crawl/segments/20070415222440
> > Generator: filtering: false
> > Generator: topN: 2147483647
> > Generator: jobtracker is 'local', generating exactly one partition.
> > Generator: Partitioning selected urls by host, for politeness.
> > Generator: done.
> > Fetcher: starting
> > Fetcher: segment: crawl/segments/20070415222440
> > Fetcher: threads: 10
> > fetching http://www.yahoo.com/
> > Fetcher: done
> > CrawlDb update: starting
> > CrawlDb update: db: crawl/crawldb
> > CrawlDb update: segments: [crawl/segments/20070415222440]
> > CrawlDb update: additions allowed: true
> > CrawlDb update: URL normalizing: true
> > CrawlDb update: URL filtering: true
> > CrawlDb update: Merging segment data into db.
> > CrawlDb update: done
> > Generator: Selecting best-scoring urls due for fetch.
> > Generator: starting
> > Generator: segment: crawl/segments/20070415222449
> > Generator: filtering: false
> > Generator: topN: 2147483647
> > Generator: jobtracker is 'local', generating exactly one partition.
> > Generator: 0 records selected for fetching, exiting ...
> > Stopping at depth=1 - no more URLs to fetch.
> > LinkDb: starting
> > LinkDb: linkdb: crawl/linkdb
> > LinkDb: URL normalize: true
> > LinkDb: URL filter: true
> > LinkDb: adding segment: crawl/segments/20070415222440
> > LinkDb: done
> > Indexer: starting
> > Indexer: linkdb: crawl/linkdb
> > Indexer: adding segment: crawl/segments/20070415222440
> > Indexing [http://www.yahoo.com/] with analyzer
> > org.apache.nutch.analysis.NutchDocumentAnalyzer@76ab2f (null)
> > Optimizing index.
> > merging segments _ram_0 (1 docs) into _0 (1 docs)
> > Indexer: done
> > Dedup: starting
> > Dedup: adding indexes in: crawl/indexes
> > Dedup: done
> > merging indexes to: crawl/index
> > Adding crawl/indexes/part-00000
> > done merging
> > crawl finished: crawl
> >
>
Re: Crawl www.yahoo.com with nutch
Posted by Meryl Silverburgh <si...@gmail.com>.
I am using 0.9 too.
I am now getting further, but I get a bunch of NullPointerException:
fetch of http://www.yahoo.com/s/557760 failed with:
java.lang.NullPointerException
fetch of http://www.yahoo.com/r/hq failed with: java.lang.NullPointerException
fetch of http://www.yahoo.com/s/557762 failed with:
java.lang.NullPointerException
On 4/15/07, songjue <so...@gmail.com> wrote:
> I try this, Nutch0.9 works just fine. What's your Nutch version?
>
>
>
> songjue
> 2007-04-16
>
>
>
> 发件人： Meryl Silverburgh
> 发送时间： 2007-04-16 11:33:05
> 收件人： nutch-user@lucene.apache.org
> 抄送：
> 主题： Crawl www.yahoo.com with nutch
>
> I setup nutch to crawl, in my input file, I only have 1 site
> "http://www.yahoo.com"
>
> $ bin/nutch crawl urls -dir crawl -depth 3
>
> and I have added 'yahoo.com' as my domain name in crawl-urlfilter.txt
>
> # accept hosts in MY.DOMAIN.NAME
> +^http://([a-zA-Z0-9]*\.)*(cnn.com|yahoo.com)/
>
> But no links is being fetched, when I change the link to www.cnn.com,
> it works. Can you please tell me what do I need to work to make
> www.yahoo.com works?
>
> $ bin/nutch crawl urls -dir crawl -depth 3
> crawl started in: crawl
> rootUrlDir = urls
> threads = 10
> depth = 3
> Injector: starting
> Injector: crawlDb: crawl/crawldb
> Injector: urlDir: urls
> Injector: Converting injected urls to crawl db entries.
> Injector: Merging injected urls into crawl db.
> Injector: done
> Generator: Selecting best-scoring urls due for fetch.
> Generator: starting
> Generator: segment: crawl/segments/20070415222440
> Generator: filtering: false
> Generator: topN: 2147483647
> Generator: jobtracker is 'local', generating exactly one partition.
> Generator: Partitioning selected urls by host, for politeness.
> Generator: done.
> Fetcher: starting
> Fetcher: segment: crawl/segments/20070415222440
> Fetcher: threads: 10
> fetching http://www.yahoo.com/
> Fetcher: done
> CrawlDb update: starting
> CrawlDb update: db: crawl/crawldb
> CrawlDb update: segments: [crawl/segments/20070415222440]
> CrawlDb update: additions allowed: true
> CrawlDb update: URL normalizing: true
> CrawlDb update: URL filtering: true
> CrawlDb update: Merging segment data into db.
> CrawlDb update: done
> Generator: Selecting best-scoring urls due for fetch.
> Generator: starting
> Generator: segment: crawl/segments/20070415222449
> Generator: filtering: false
> Generator: topN: 2147483647
> Generator: jobtracker is 'local', generating exactly one partition.
> Generator: 0 records selected for fetching, exiting ...
> Stopping at depth=1 - no more URLs to fetch.
> LinkDb: starting
> LinkDb: linkdb: crawl/linkdb
> LinkDb: URL normalize: true
> LinkDb: URL filter: true
> LinkDb: adding segment: crawl/segments/20070415222440
> LinkDb: done
> Indexer: starting
> Indexer: linkdb: crawl/linkdb
> Indexer: adding segment: crawl/segments/20070415222440
> Indexing [http://www.yahoo.com/] with analyzer
> org.apache.nutch.analysis.NutchDocumentAnalyzer@76ab2f (null)
> Optimizing index.
> merging segments _ram_0 (1 docs) into _0 (1 docs)
> Indexer: done
> Dedup: starting
> Dedup: adding indexes in: crawl/indexes
> Dedup: done
> merging indexes to: crawl/index
> Adding crawl/indexes/part-00000
> done merging
> crawl finished: crawl
>
Re: Crawl www.yahoo.com with nutch
Posted by songjue <so...@gmail.com>.
I try this, Nutch0.9 works just fine. What's your Nutch version?
songjue
2007-04-16
发件人： Meryl Silverburgh
发送时间： 2007-04-16 11:33:05
收件人： nutch-user@lucene.apache.org
抄送：
主题： Crawl www.yahoo.com with nutch
I setup nutch to crawl, in my input file, I only have 1 site
"http://www.yahoo.com"
$ bin/nutch crawl urls -dir crawl -depth 3
and I have added 'yahoo.com' as my domain name in crawl-urlfilter.txt
# accept hosts in MY.DOMAIN.NAME
+^http://([a-zA-Z0-9]*\.)*(cnn.com|yahoo.com)/
But no links is being fetched, when I change the link to www.cnn.com,
it works. Can you please tell me what do I need to work to make
www.yahoo.com works?
$ bin/nutch crawl urls -dir crawl -depth 3
crawl started in: crawl
rootUrlDir = urls
threads = 10
depth = 3
Injector: starting
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: crawl/segments/20070415222440
Generator: filtering: false
Generator: topN: 2147483647
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls by host, for politeness.
Generator: done.
Fetcher: starting
Fetcher: segment: crawl/segments/20070415222440
Fetcher: threads: 10
fetching http://www.yahoo.com/
Fetcher: done
CrawlDb update: starting
CrawlDb update: db: crawl/crawldb
CrawlDb update: segments: [crawl/segments/20070415222440]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: crawl/segments/20070415222449
Generator: filtering: false
Generator: topN: 2147483647
Generator: jobtracker is 'local', generating exactly one partition.
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=1 - no more URLs to fetch.
LinkDb: starting
LinkDb: linkdb: crawl/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: adding segment: crawl/segments/20070415222440
LinkDb: done
Indexer: starting
Indexer: linkdb: crawl/linkdb
Indexer: adding segment: crawl/segments/20070415222440
Indexing [http://www.yahoo.com/] with analyzer
org.apache.nutch.analysis.NutchDocumentAnalyzer@76ab2f (null)
Optimizing index.
merging segments _ram_0 (1 docs) into _0 (1 docs)
Indexer: done
Dedup: starting
Dedup: adding indexes in: crawl/indexes
Dedup: done
merging indexes to: crawl/index
Adding crawl/indexes/part-00000
done merging
crawl finished: crawl