You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by sumarlidason <su...@gmail.com> on 2012/10/19 17:12:59 UTC
nutch/hadoop/solr
Good Morning,
So I made progress since my last post. yay!
Next problem, I am receiving this error when attempting this command:
./nutch crawl urls -dir crawl -solr http://10.100.220.220:8983/solr/ -depth
3 -topN 5
Exception in thread "main" org.apache.gora.util.GoraException:
java.io.IOException: java.lang.ClassNotFoundException:
org.hsqldb.jdbc.JDBCDriver
...
This looks like a DB connector exception, but I haven't specified a DB
connection..? I have been using 1.5.1 locally, and I am trying to transition
to 2.1 hadoop.
The command that the nutch script executes:
exec hadoop jar /root/Downloads/apache-nutch-2.1/build/apache-nutch-2.1.job
org.apache.nutch.crawl.Crawler urls -dir crawl -solr
http://10.100.220.220:8983/solr/ -depth 3 -topN 5
and the hdfs contains..
drwxr-xr-x - root wheel 26 2012-10-19 09:47 /user/root/urls
-rw-r--r-- 1 root wheel 21 2012-10-19 09:47
/user/root/urls/seed.txt
again, any help, very much appreciated
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: nutch/hadoop/solr
Posted by j....@thomsonreuters.com.
If it is the ID field on MySQL let me share my experience.
I did not change that for a reason. The ID field is a primary key and has a rather small size limit due to MySQL limitations (I think it is 1000 bytes). Even for Japanese language sites where I do most of my crawling all of the URLs I am interested in are Ascii so in my case there wasn't the need.
If you do need to change it (for non-English URLs) use varchar not char because if you use char with UTF8 or utf8mb4 you will allocate 3 or 4 bytes respectively per character regardless of what the character actually needs and you will find long urls causing errors as they exceed the limit you set or the 1000 byte absolute limit. If you use varchar it will allocate the needed amount of bytes (1 per English character more for characters in other languages). That doesn't completely rule out running into a URL that takes more than 1000 bytes but makes it whole lot less likely as the majority of urls use English characters.
I will look at changing it in my example but it may be a bit later as I think it needs some testing to make sure I don't cause problems.
If you are crawling websites with UR
-----Original Message-----
From: sumarlidason [mailto:sumarlidason@gmail.com]
Sent: Thursday, October 25, 2012 12:48 AM
To: user@nutch.apache.org
Subject: RE: nutch/hadoop/solr
err, manager pointed out that its the ID field complaining now.. so attempting to change the collation there as well.
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015626.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: nutch/hadoop/solr
Posted by sumarlidason <su...@gmail.com>.
I sent an email to Lewis with the following:
When using MYSQL 5.5 w/ utf8mb4 the id column, a primary key, is restricted
to 190 characters, which is insufficient.
- Does the ID need to be the primary key? can it just be unique? I don't
think unique has a restriction on character length, but I haven't confirmed
this yet.
- Could we use a hash of the url as the id instead of the url itself?
If no one else is having this issue, i must be doing something wrong.
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015730.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: nutch/hadoop/solr
Posted by sumarlidason <su...@gmail.com>.
err, manager pointed out that its the ID field complaining now.. so
attempting to change the collation there as well.
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015626.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: nutch/hadoop/solr
Posted by sumarlidason <su...@gmail.com>.
James,
I ran a crawl using your db modifications, I'm still receiving the
exceptions breaking the crawl. Its hard to believe I'm the only one
receiving these errors.
java.io.IOException: java.sql.BatchUpdateException: Incorrect string value:
'\xE2\x80\x8Butu...' for column 'id' at row 1
at org.apache.gora.sql.store.SqlStore.flush(SqlStore.java:340)
at
org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:65)
Oddly now this time, it seems to be a 3 byte character, \xE2\x80\x8B ...
playing with these characters in osx is as easy as 123, and not so much in
windows. :(
My next thought is to tear open gora and do something code side to deal with
these special characters. I looked at the SqlStore.flush() function last
night, but couldn't get gora_sqlstore to compile : some unresolved
dependencies.
any additional thoughts?
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015606.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: nutch/hadoop/solr
Posted by j....@thomsonreuters.com.
Okay, sounds like you may actually need it. I've updated the information at http://nlp.solutions.asia/?p=180 to use utf8mb4. If you could use that with MySQL 5.5 or above and see if it helps. It is changed in three places -- the db server config, the db creation and the table creation.
-----Original Message-----
From: sumarlidason [mailto:sumarlidason@gmail.com]
Sent: Wednesday, October 24, 2012 9:31 AM
To: user@nutch.apache.org
Subject: RE: nutch/hadoop/solr
Actually, that is the tutorial I followed.
I'm still getting these errors.. this string, \xF0\x9F\x92\x83, is actually this character:
I assume thats where the issue is. However I am unable to reproduce the error when manually inserting via /usr/bin/mysql.
I read this article,
http://mzsanford.wordpress.com/2010/12/28/mysql-and-unicode/, he suggests that utf8_bin might resolve the issue. Other forums suggest that even though the default charset is set, the column charset has to be specifically set as well.
I can't get passed the fact that MySQL pre 5.5 is only storing 1-3Bytes UTF instead of 1-4Bytes.
j.sullivan wrote
> Sumarlidason
>
> Hi
>
> The need to use utf8mb4 for web crawling should be fairly rare. If you
> are using MySQL 5.5 or later and have a set up like this
> http://nlp.solutions.asia/?p=180 you should be fine.
>
> James
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015480.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: nutch/hadoop/solr
Posted by sumarlidason <su...@gmail.com>.
Actually, that is the tutorial I followed.
I'm still getting these errors.. this string, \xF0\x9F\x92\x83, is actually
this character: 💃
I assume thats where the issue is. However I am unable to reproduce the
error when manually inserting via /usr/bin/mysql.
I read this article,
http://mzsanford.wordpress.com/2010/12/28/mysql-and-unicode/, he suggests
that utf8_bin might resolve the issue. Other forums suggest that even though
the default charset is set, the column charset has to be specifically set as
well.
I can't get passed the fact that MySQL pre 5.5 is only storing 1-3Bytes UTF
instead of 1-4Bytes.
j.sullivan wrote
> Sumarlidason
>
> Hi
>
> The need to use utf8mb4 for web crawling should be fairly rare. If you are
> using MySQL 5.5 or later and have a set up like this
> http://nlp.solutions.asia/?p=180 you should be fine.
>
> James
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015480.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: nutch/hadoop/solr
Posted by j....@thomsonreuters.com.
Sumarlidason
Hi
The need to use utf8mb4 for web crawling should be fairly rare. If you are using MySQL 5.5 or later and have a set up like this http://nlp.solutions.asia/?p=180 you should be fine.
James
-----Original Message-----
From: sumarlidason [mailto:sumarlidason@gmail.com]
Sent: Wednesday, October 24, 2012 3:22 AM
To: user@nutch.apache.org
Subject: Re: nutch/hadoop/solr
Running a "deeper" crawl revealed another problem, research has lead me to beleive I need to upgrade to latest MySQL Server Version and use utf8mb4 encoding.
Does this sound 'Etch-a-Sketchy' to you?
java.io.IOException: java.sql.BatchUpdateException: Incorrect string value:
'\xF0\x9F\x92\x83Li...' for column 'text' at row 1
at org.apache.gora.sql.store.SqlStore.flush(SqlStore.java:340)
at org.apache.gora.sql.store.SqlStore.close(SqlStore.java:185)
at
org.apache.gora.mapreduce.GoraRecordWriter.close(GoraRecordWriter.java:55)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:540)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:649)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: java.sql.BatchUpdateException: Incorrect string value:
'\xF0\x9F\x92\x83Li...' for column 'text' at row 1
at com.mysql.jdbc.PreparedStatement.exe
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015415.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: nutch/hadoop/solr
Posted by sumarlidason <su...@gmail.com>.
Running a "deeper" crawl revealed another problem, research has lead me to
beleive I need to upgrade to latest MySQL Server Version and use utf8mb4
encoding.
Does this sound 'Etch-a-Sketchy' to you?
java.io.IOException: java.sql.BatchUpdateException: Incorrect string value:
'\xF0\x9F\x92\x83Li...' for column 'text' at row 1
at org.apache.gora.sql.store.SqlStore.flush(SqlStore.java:340)
at org.apache.gora.sql.store.SqlStore.close(SqlStore.java:185)
at
org.apache.gora.mapreduce.GoraRecordWriter.close(GoraRecordWriter.java:55)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:540)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:649)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: java.sql.BatchUpdateException: Incorrect string value:
'\xF0\x9F\x92\x83Li...' for column 'text' at row 1
at com.mysql.jdbc.PreparedStatement.exe
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015415.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: nutch/hadoop/solr
Posted by sumarlidason <su...@gmail.com>.
Additionally,
>From reading another post, I attempted to run the indexer manually, with
success.
sudo -u mapred hadoop jar apache-nutch-2.1.job
org.apache.nutch.indexer.solr.SolrIndexerJob
http://10.100.220.220:8983/solr/ -reindex
However, still would like to correct the problem above..
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015388.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: nutch/hadoop/solr
Posted by sumarlidason <su...@gmail.com>.
Alright Lewis,
Leaps and bounds over yesterday's progress. I've abandon the idea of using
HBASE for now. I setup mySQL database. Launch successfully, and now fail @
IndexerJob.createIndexJob()
*gora.properties*
gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
gora.sqlstore.jdbc.url=jdbc:mysql://10.100.220.220:3306/nutch?createDatabaseIfNotExist=true
gora.sqlstore.jdbc.user=root
gora.sqlstore.jdbc.password=pw
*nutch-site.xml*
<configuration>
<property>
<name>http.agent.name</name>
<value>The Nutchess</value>
</property>
<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
<description>The character encoding to fall back to when no other
information
is available</description>
</property>
<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.sql.store.SqlStore</value>
<description>The Gora DataStore class for storing and retrieving data.
Currently the following stores are available: ..
</description>
</property>
</configuration>
[root@hdpjt01 build]# sudo -u mapred hadoop jar apache-nutch-2.1.job
org.apache.nutch.crawl.Crawler urls -solr http://10.100.220.220:8983/solr/
-depth 3 -topN 5
12/10/23 10:31:17 INFO input.FileInputFormat: Total input paths to process :
1
12/10/23 10:31:17 WARN snappy.LoadSnappy: Snappy native library is available
12/10/23 10:31:17 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
12/10/23 10:31:17 INFO snappy.LoadSnappy: Snappy native library loaded
12/10/23 10:31:18 INFO mapred.JobClient: Running job: job_201210221719_0006
12/10/23 10:31:19 INFO mapred.JobClient: map 0% reduce 0%
12/10/23 10:31:28 INFO mapred.JobClient: map 100% reduce 0%
12/10/23 10:31:29 INFO mapred.JobClient: Job complete: job_201210221719_0006
*... Several more jobs ...*
12/10/23 10:36:02 INFO mapred.JobClient: Job complete: job_201210221719_0018
12/10/23 10:36:02 INFO mapred.JobClient: Counters: 24
12/10/23 10:36:02 INFO mapred.JobClient: Job Counters
12/10/23 10:36:02 INFO mapred.JobClient: Launched reduce tasks=1
12/10/23 10:36:02 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=7562
12/10/23 10:36:02 INFO mapred.JobClient: Total time spent by all reduces
waiting after reserving slots (ms)=0
12/10/23 10:36:02 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0
12/10/23 10:36:02 INFO mapred.JobClient: Launched map tasks=1
12/10/23 10:36:02 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=9683
12/10/23 10:36:02 INFO mapred.JobClient: FileSystemCounters
12/10/23 10:36:02 INFO mapred.JobClient: FILE_BYTES_READ=280978
12/10/23 10:36:02 INFO mapred.JobClient: HDFS_BYTES_READ=1078
12/10/23 10:36:02 INFO mapred.JobClient: FILE_BYTES_WRITTEN=738512
12/10/23 10:36:02 INFO mapred.JobClient: Map-Reduce Framework
12/10/23 10:36:02 INFO mapred.JobClient: Map input records=688
12/10/23 10:36:02 INFO mapred.JobClient: Reduce shuffle bytes=280978
12/10/23 10:36:02 INFO mapred.JobClient: Spilled Records=3078
12/10/23 10:36:02 INFO mapred.JobClient: Map output bytes=277753
12/10/23 10:36:02 INFO mapred.JobClient: CPU time spent (ms)=9830
12/10/23 10:36:02 INFO mapred.JobClient: Total committed heap usage
(bytes)=891486208
12/10/23 10:36:02 INFO mapred.JobClient: Combine input records=0
12/10/23 10:36:02 INFO mapred.JobClient: SPLIT_RAW_BYTES=1078
12/10/23 10:36:02 INFO mapred.JobClient: Reduce input records=1539
12/10/23 10:36:02 INFO mapred.JobClient: Reduce input groups=820
12/10/23 10:36:02 INFO mapred.JobClient: Combine output records=0
12/10/23 10:36:02 INFO mapred.JobClient: Physical memory (bytes)
snapshot=949526528
12/10/23 10:36:02 INFO mapred.JobClient: Reduce output records=820
12/10/23 10:36:02 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=3199545344
12/10/23 10:36:02 INFO mapred.JobClient: Map output records=1539
Exception in thread "main" java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:460)
at
org.apache.nutch.indexer.IndexerJob.createIndexJob(IndexerJob.java:128)
at
org.apache.nutch.indexer.solr.SolrIndexerJob.run(SolrIndexerJob.java:44)
at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68)
at org.apache.nutch.crawl.Crawler.run(Crawler.java:192)
at org.apache.nutch.crawl.Crawler.run(Crawler.java:250)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawler.main(Crawler.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4015379.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: nutch/hadoop/solr
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi,
On Fri, Oct 19, 2012 at 6:23 PM, sumarlidason <su...@gmail.com> wrote:
> So, I made some changes to gora.properties, and now im getting null pointer
> exception..
Do you wish to detail when and how you are getting this Exception?
> It almost seems to me at this point nutch needs a DB for its index, instead
> of working on disk? is this correct? ..
yes
> and then when specifying the -solr
> parameter the data is copied to solr's index when finished.. is this correct
> assumption?
In short yes. If you are looking to transition to 2.X I would direct
you to the tutorials on the Nutch wiki.
hth
Lewis
Re: nutch/hadoop/solr
Posted by sumarlidason <su...@gmail.com>.
So, I made some changes to gora.properties, and now im getting null pointer
exception..
It almost seems to me at this point nutch needs a DB for its index, instead
of working on disk? is this correct? .. and then when specifying the -solr
parameter the data is copied to solr's index when finished.. is this correct
assumption?
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-hadoop-solr-tp4014761p4014797.html
Sent from the Nutch - User mailing list archive at Nabble.com.