You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Armel T. Nene" <ar...@idna-solutions.com> on 2007/01/23 19:54:03 UTC
Exception java.lang.ArithmeticException
Hi guys,
I'm new to the mailing list therefore I apologize if this posting is not
meant to be in here.
I have been working with Nutch now for quite awhile. I am currently building
a tool that can read and retrieve Nutch CrawlDb information. What am I here
if I'm talking about Nutch? As I understand Nutch uses the Hadoop file
system and many of its API. The CrawlDb is actually an hadoop MapFile
object. I understand that the hadoop filesystem doesn't support update as
it's only write-once.
My problem is when I run my program I get the following errors:
07/01/23 18:33:56 INFO conf.Configuration: parsing
jar:file:/C:/nutch-0.8.2-dev/lib/hadoop-0.4.0-patched.jar!/hadoop-default.xm
l
07/01/23 18:33:56 INFO conf.Configuration: parsing
jar:file:/C:/nutch-0.8.2-dev/nutch-0.8.2-dev.jar!/nutch-default.xml
07/01/23 18:33:56 INFO conf.Configuration: parsing
jar:file:/C:/nutch-0.8.2-dev/nutch-0.8.2-dev.jar!/nutch-site.xml
Exception in thread "main" java.lang.ArithmeticException: / by zero
at
org.apache.hadoop.mapred.lib.HashPartitioner.getPartition(HashPartitioner.ja
va:33)
at
org.apache.hadoop.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.ja
va:88)
at org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:321)
at
com.idna.indexer.resource.SystemMonitor.main(SystemMonitor.java:188)
I'll be looking forward to your helping reply.
Best regards,
Armel Nene
-------------------------------------------------
Armel T. Nene
iDNA Solutions
Tel: +44 (207) 257 6124
Mobile: +44 (788) 695 0483
<http://blog.idna-solutions.com/> http://blog.idna-solutions.com
RE: Exception java.lang.ArithmeticException
Posted by "Armel T. Nene" <ar...@idna-solutions.com>.
Doug,
This an extract from the same crawldb, I am running application from outside
of Nutch. Does hadoop checks for anything else or is there a better to
iterate over the values in crawldb without going through hadoop (This might
actually be a question for the nutch community).
Here is an extract from my crawldb
http://dmoz.org/ Version: 4
Status: 2 (DB_fetched)
Fetch time: Thu Feb 22 12:44:05 GMT 2007
Modified time: Thu Jan 01 01:00:00 GMT 1970
Retries since fetch: 0
Retry interval: 30.0 days
Score: 1.0323955
Signature: f4c14c46074b66aad8829b8aa84cd636
Metadata: null
http://dmoz.org/Arts/ Version: 4
Status: 2 (DB_fetched)
Fetch time: Thu Feb 22 12:45:43 GMT 2007
Modified time: Thu Jan 01 01:00:00 GMT 1970
Retries since fetch: 0
Retry interval: 30.0 days
Score: 0.013471641
Signature: fe52a0bcb1071070689d0f661c168648
Metadata: null
Thanks
Armel
-------------------------------------------------
Armel T. Nene
iDNA Solutions
Tel: +44 (207) 257 6124
Mobile: +44 (788) 695 0483
http://blog.idna-solutions.com
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: 23 January 2007 19:08
To: hadoop-dev@lucene.apache.org
Subject: Re: Exception java.lang.ArithmeticException
Armel T. Nene wrote:
> My problem is when I run my program I get the following errors:
> Exception in thread "main" java.lang.ArithmeticException: / by zero
>
> at
>
org.apache.hadoop.mapred.lib.HashPartitioner.getPartition(HashPartitioner.ja
> va:33)
>
> at
>
org.apache.hadoop.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.ja
> va:88)
>
> at
org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:321)
>
> at
It looks like the crawl db path contains no files. Probably
MapFileOutputFormat#getReaders() should throw an exception when no paths
are listed (if 'names', on line 68, has length zero). That might make
this a little easier to debug.
So please check the path you're passing for the crawl db.
Doug
Re: Exception java.lang.ArithmeticException
Posted by Doug Cutting <cu...@apache.org>.
Armel T. Nene wrote:
> My problem is when I run my program I get the following errors:
> Exception in thread "main" java.lang.ArithmeticException: / by zero
>
> at
> org.apache.hadoop.mapred.lib.HashPartitioner.getPartition(HashPartitioner.ja
> va:33)
>
> at
> org.apache.hadoop.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.ja
> va:88)
>
> at org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:321)
>
> at
It looks like the crawl db path contains no files. Probably
MapFileOutputFormat#getReaders() should throw an exception when no paths
are listed (if 'names', on line 68, has length zero). That might make
this a little easier to debug.
So please check the path you're passing for the crawl db.
Doug
Re: Exception java.lang.ArithmeticException
Posted by Doug Cutting <cu...@apache.org>.
Armel T. Nene wrote:
> I am running the system locally through Nutch. Nutch also run on top of
> hadoop. All I am trying to access the file through the following method:
>
> MapFileOutputFormat.getEntry(readers, new HashPartitioner(), key, val);
>
> Nutch runs successfully and loads the Hadoop configuration files and
> anything needed. Can you let me know how to set reduce to a different value
> than 0.
This isn't actually under reduce. The problem is that the 'readers'
array above is empty. This is returned by
MapFileOutputFormat#getReaders() when passed an inappropriate path.
Doug
RE: Exception java.lang.ArithmeticException
Posted by "Armel T. Nene" <ar...@idna-solutions.com>.
I am running the system locally through Nutch. Nutch also run on top of
hadoop. All I am trying to access the file through the following method:
MapFileOutputFormat.getEntry(readers, new HashPartitioner(), key, val);
Nutch runs successfully and loads the Hadoop configuration files and
anything needed. Can you let me know how to set reduce to a different value
than 0.
Thanks,
Armel
-------------------------------------------------
Armel T. Nene
iDNA Solutions
Tel: +44 (207) 257 6124
Mobile: +44 (788) 695 0483
http://blog.idna-solutions.com
-----Original Message-----
From: Owen O'Malley [mailto:owen@yahoo-inc.com]
Sent: 23 January 2007 19:01
To: hadoop-dev@lucene.apache.org
Subject: Re: Exception java.lang.ArithmeticException
On Jan 23, 2007, at 10:54 AM, Armel T. Nene wrote:
> I'm new to the mailing list therefore I apologize if this posting
> is not
> meant to be in here.
Your post is good.
> My problem is when I run my program I get the following errors:
>
>
>
> 07/01/23 18:33:56 INFO conf.Configuration: parsing
> jar:file:/C:/nutch-0.8.2-dev/lib/hadoop-0.4.0-patched.jar!/hadoop-
> default.xm
> l
>
> 07/01/23 18:33:56 INFO conf.Configuration: parsing
> jar:file:/C:/nutch-0.8.2-dev/nutch-0.8.2-dev.jar!/nutch-default.xml
>
> 07/01/23 18:33:56 INFO conf.Configuration: parsing
> jar:file:/C:/nutch-0.8.2-dev/nutch-0.8.2-dev.jar!/nutch-site.xml
>
> Exception in thread "main" java.lang.ArithmeticException: / by zero
>
> at
> org.apache.hadoop.mapred.lib.HashPartitioner.getPartition
> (HashPartitioner.ja
> va:33)
>
> at
> org.apache.hadoop.mapred.MapFileOutputFormat.getEntry
> (MapFileOutputFormat.ja
> va:88)
>
> at org.apache.nutch.crawl.CrawlDbReader.get
> (CrawlDbReader.java:321)
>
> at
> com.idna.indexer.resource.SystemMonitor.main(SystemMonitor.java:188)
It looks like you are running distributed and have the number of
reduces set to 0. You should probably set it to 2*numOfNodes.
-- Owen
Re: Exception java.lang.ArithmeticException
Posted by Owen O'Malley <ow...@yahoo-inc.com>.
On Jan 23, 2007, at 10:54 AM, Armel T. Nene wrote:
> I'm new to the mailing list therefore I apologize if this posting
> is not
> meant to be in here.
Your post is good.
> My problem is when I run my program I get the following errors:
>
>
>
> 07/01/23 18:33:56 INFO conf.Configuration: parsing
> jar:file:/C:/nutch-0.8.2-dev/lib/hadoop-0.4.0-patched.jar!/hadoop-
> default.xm
> l
>
> 07/01/23 18:33:56 INFO conf.Configuration: parsing
> jar:file:/C:/nutch-0.8.2-dev/nutch-0.8.2-dev.jar!/nutch-default.xml
>
> 07/01/23 18:33:56 INFO conf.Configuration: parsing
> jar:file:/C:/nutch-0.8.2-dev/nutch-0.8.2-dev.jar!/nutch-site.xml
>
> Exception in thread "main" java.lang.ArithmeticException: / by zero
>
> at
> org.apache.hadoop.mapred.lib.HashPartitioner.getPartition
> (HashPartitioner.ja
> va:33)
>
> at
> org.apache.hadoop.mapred.MapFileOutputFormat.getEntry
> (MapFileOutputFormat.ja
> va:88)
>
> at org.apache.nutch.crawl.CrawlDbReader.get
> (CrawlDbReader.java:321)
>
> at
> com.idna.indexer.resource.SystemMonitor.main(SystemMonitor.java:188)
It looks like you are running distributed and have the number of
reduces set to 0. You should probably set it to 2*numOfNodes.
-- Owen