You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by István <le...@gmail.com> on 2017/10/11 13:05:50 UTC
Indexing files from HDFS
Hi,
I have Solr 4.10.3 part of a CDH5 installation and I would like to index
huge amount of CSV files on HDFS. I was wondering what is the best way of
doing that.
Here is the current approach:
data.csv:
id, fruit
10, apple
20, orange
Indexing with the following command using search-mr-1.0.0-cdh5.11.1-job.jar
hadoop --config /etc/hadoop/conf.cloudera.yarn jar \
/opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-1.0.0-cdh5.11.1-job.jar
\
org.apache.solr.hadoop.MapReduceIndexerTool \
-D 'mapred.child.java.opts=-Xmx500m' --log4j \
/opt/cloudera/parcels/CDH/share/doc/search/examples/solr-nrt/log4j.properties
--morphline-file \
/home/user/readCSV.conf \
--output-dir hdfs://name-node.server.com:8020/user/solr/output --verbose
--go-live \
--zk-host name-node.server.com:2181/solr --collection collection0 \
hdfs://name-node.server.com:8020/user/solr/input
This leads to the following exception:
2219 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Indexing 1
files using 1 real mappers into 1 reducers
Error: java.io.IOException: Batch Write Failure
at org.apache.solr.hadoop.BatchWriter.throwIf(BatchWriter.java:239)
..
Caused by: org.apache.solr.common.SolrException: ERROR: [doc=100] unknown
field 'file_path'
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:185)
at
org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:78)
It appears to me that the schema does not have file_path. The collection is
created through Hue and it properly identifies the two fields id and fruit.
I found out that the search-mr tool has the following code that references
the file_path:
https://github.com/cloudera/search/blob/cdh5-1.0.0_5.2.0/search-mr/src/main/java/org/apache/solr/hadoop/HdfsFileFieldNames.java#L30
I am not sure what to do in order to be able to index files on HDFS. I have
two guesses:
- add the fields definied in the search tool to the schema when I create it
(not sure how that work through Hue)
- disable the HDFS meatadata insertion when inserting data
Has anybody seen this before?
Thanks,
Istvan
--
the sun shines for all
Re: Indexing files from HDFS
Posted by Shawn Heisey <el...@elyograg.org>.
On 10/12/2017 2:04 AM, István wrote:
> The question is not about Hue but about why file_path is in the schema for
> HDFS files when using search-mr. I am wondering what is the standard way of
> indexing files on HDFS.
The error in your original post indicates that at least one document in
the update request contains a "file_path" field, but the active schema
on the Solr index does NOT have that field, so Solr is not able to
handle the indexing request.
It appears that you are using Cloudera software to do the indexing. If
you cannot tell why the indexing requests have that field, then you will
need to talk to Cloudera about how their software works.
One idea that might work is to add the file_path field to your schema
with a correct type so the indexing requests that are being sent will be
handled correctly.
Thanks,
Shawn
Re: Indexing files from HDFS
Posted by István <le...@gmail.com>.
Hi Erik,
The question is not about Hue but about why file_path is in the schema for
HDFS files when using search-mr. I am wondering what is the standard way of
indexing files on HDFS.
THanks,
Istvan
On Wed, Oct 11, 2017 at 4:53 PM, Erick Erickson <er...@gmail.com>
wrote:
> You probably get much more informed responses from
> the Cloudera folks, especially about Hue.
>
> Best,
> Erick
>
> On Wed, Oct 11, 2017 at 6:05 AM, István <le...@gmail.com> wrote:
> > Hi,
> >
> > I have Solr 4.10.3 part of a CDH5 installation and I would like to index
> > huge amount of CSV files on HDFS. I was wondering what is the best way of
> > doing that.
> >
> > Here is the current approach:
> >
> > data.csv:
> >
> > id, fruit
> > 10, apple
> > 20, orange
> >
> > Indexing with the following command using search-mr-1.0.0-cdh5.11.1-job.
> jar
> >
> > hadoop --config /etc/hadoop/conf.cloudera.yarn jar \
> > /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-1.0.
> 0-cdh5.11.1-job.jar
> > \
> > org.apache.solr.hadoop.MapReduceIndexerTool \
> > -D 'mapred.child.java.opts=-Xmx500m' --log4j \
> > /opt/cloudera/parcels/CDH/share/doc/search/examples/
> solr-nrt/log4j.properties
> > --morphline-file \
> > /home/user/readCSV.conf \
> > --output-dir hdfs://name-node.server.com:8020/user/solr/output --verbose
> > --go-live \
> > --zk-host name-node.server.com:2181/solr --collection collection0 \
> > hdfs://name-node.server.com:8020/user/solr/input
> >
> > This leads to the following exception:
> >
> > 2219 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool -
> Indexing 1
> > files using 1 real mappers into 1 reducers
> > Error: java.io.IOException: Batch Write Failure
> > at org.apache.solr.hadoop.BatchWriter.throwIf(
> BatchWriter.java:239)
> > ..
> > Caused by: org.apache.solr.common.SolrException: ERROR: [doc=100]
> unknown
> > field 'file_path'
> > at
> > org.apache.solr.update.DocumentBuilder.toDocument(
> DocumentBuilder.java:185)
> > at
> > org.apache.solr.update.AddUpdateCommand.getLuceneDocument(
> AddUpdateCommand.java:78)
> >
> > It appears to me that the schema does not have file_path. The collection
> is
> > created through Hue and it properly identifies the two fields id and
> fruit.
> > I found out that the search-mr tool has the following code that
> references
> > the file_path:
> >
> > https://github.com/cloudera/search/blob/cdh5-1.0.0_5.2.0/
> search-mr/src/main/java/org/apache/solr/hadoop/HdfsFileFieldNames.java#L30
> >
> > I am not sure what to do in order to be able to index files on HDFS. I
> have
> > two guesses:
> >
> > - add the fields definied in the search tool to the schema when I create
> it
> > (not sure how that work through Hue)
> > - disable the HDFS meatadata insertion when inserting data
> >
> > Has anybody seen this before?
> >
> > Thanks,
> > Istvan
> >
> >
> >
> >
> > --
> > the sun shines for all
>
--
the sun shines for all
Re: Indexing files from HDFS
Posted by Erick Erickson <er...@gmail.com>.
You probably get much more informed responses from
the Cloudera folks, especially about Hue.
Best,
Erick
On Wed, Oct 11, 2017 at 6:05 AM, István <le...@gmail.com> wrote:
> Hi,
>
> I have Solr 4.10.3 part of a CDH5 installation and I would like to index
> huge amount of CSV files on HDFS. I was wondering what is the best way of
> doing that.
>
> Here is the current approach:
>
> data.csv:
>
> id, fruit
> 10, apple
> 20, orange
>
> Indexing with the following command using search-mr-1.0.0-cdh5.11.1-job.jar
>
> hadoop --config /etc/hadoop/conf.cloudera.yarn jar \
> /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-1.0.0-cdh5.11.1-job.jar
> \
> org.apache.solr.hadoop.MapReduceIndexerTool \
> -D 'mapred.child.java.opts=-Xmx500m' --log4j \
> /opt/cloudera/parcels/CDH/share/doc/search/examples/solr-nrt/log4j.properties
> --morphline-file \
> /home/user/readCSV.conf \
> --output-dir hdfs://name-node.server.com:8020/user/solr/output --verbose
> --go-live \
> --zk-host name-node.server.com:2181/solr --collection collection0 \
> hdfs://name-node.server.com:8020/user/solr/input
>
> This leads to the following exception:
>
> 2219 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Indexing 1
> files using 1 real mappers into 1 reducers
> Error: java.io.IOException: Batch Write Failure
> at org.apache.solr.hadoop.BatchWriter.throwIf(BatchWriter.java:239)
> ..
> Caused by: org.apache.solr.common.SolrException: ERROR: [doc=100] unknown
> field 'file_path'
> at
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:185)
> at
> org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:78)
>
> It appears to me that the schema does not have file_path. The collection is
> created through Hue and it properly identifies the two fields id and fruit.
> I found out that the search-mr tool has the following code that references
> the file_path:
>
> https://github.com/cloudera/search/blob/cdh5-1.0.0_5.2.0/search-mr/src/main/java/org/apache/solr/hadoop/HdfsFileFieldNames.java#L30
>
> I am not sure what to do in order to be able to index files on HDFS. I have
> two guesses:
>
> - add the fields definied in the search tool to the schema when I create it
> (not sure how that work through Hue)
> - disable the HDFS meatadata insertion when inserting data
>
> Has anybody seen this before?
>
> Thanks,
> Istvan
>
>
>
>
> --
> the sun shines for all