You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sujeet Singh <Su...@cloudmoyo.com> on 2018/06/12 11:55:47 UTC
Issues while running map-reduce index tool.
Hi Team,
I am trying to index a document from HDFS in version solr 4.9 and getting below error:
Command used by me :
QUEUE_NAME=default
MORPHLINE_CONF=/home/sshuser/solrruncls/morphline_retail_tr_2017.conf
OUTPUT_DIR=adl:///solr
ZK_HOST=zk0-hnrhba.thxycti2fi4ejjab25g3ibtkog.gx.internal.cloudapp.net:2181,zk2-hnrhba.thxycti2fi4ejjab25g3ibtkog.gx.internal.cloudapp.net:2181
SOLR_URI=hn0-hnrhba.thxycti2fi4ejjab25g3ibtkog.gx.internal.cloudapp.net:8983/solr
INPUT_DIR=adl:///solrinput
source /usr/hdp/current/solr/example/scripts/map-reduce/set-map-reduce-classpath.sh
hadoop jar /usr/hdp/current/solr/dist/solr-map-reduce-*.jar -D 'mapred.child.java.opts=-Xmx500m' -libjars "$HADOOP_LIBJAR" --morphline-file $MORPHLINE_CONF --zk-host $ZK_HOST --output-dir $OUTPUT_DIR --collection collection1 --go-live --verbose $INPUT_DIR
Error: java.lang.IllegalStateException: Failed to initialize record writer for org.apache.solr.hadoop.MapReduceIndexerTool/MorphlineMapper, attempt_1528778576449_0013_r_000000_5
at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:127)
at org.apache.solr.hadoop.SolrOutputFormat.getRecordWriter(SolrOutputFormat.java:162)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:541)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: org.apache.solr.common.SolrException: Unable to create core: core1
at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:911)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:568)
at org.apache.solr.hadoop.SolrRecordWriter.createEmbeddedSolrServer(SolrRecordWriter.java:171)
at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:120)
... 9 more
Caused by: org.apache.solr.common.SolrException: You must set the HdfsDirectoryFactory param solr.hdfs.home for relative dataDir paths to work
at org.apache.solr.core.HdfsDirectoryFactory.getDataHome(HdfsDirectoryFactory.java:263)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:696)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:643)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556)
... 11 more
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143.
18/06/12 11:53:36 INFO mapreduce.Job: Task Id : attempt_1528778576449_0013_r_000000_6, Status : FAILED
Error: java.lang.IllegalStateException: Failed to initialize record writer for org.apache.solr.hadoop.MapReduceIndexerTool/MorphlineMapper, attempt_1528778576449_0013_r_000000_6
at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:127)
at org.apache.solr.hadoop.SolrOutputFormat.getRecordWriter(SolrOutputFormat.java:162)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:541)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: org.apache.solr.common.SolrException: Unable to create core: core1
at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:911)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:568)
at org.apache.solr.hadoop.SolrRecordWriter.createEmbeddedSolrServer(SolrRecordWriter.java:171)
at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:120)
... 9 more
Caused by: org.apache.solr.common.SolrException: You must set the HdfsDirectoryFactory param solr.hdfs.home for relative dataDir paths to work
at org.apache.solr.core.HdfsDirectoryFactory.getDataHome(HdfsDirectoryFactory.java:263)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:696)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:643)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556)
... 11 more
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143.
Regards,
________________________________
Sujeet Singh | Sr. Software Analyst | cloudmoyo | E. sujeet.singh@cloudmoyo.com<ma...@cloudmoyo.com> | M. +91 9860586055
[CloudMoyo Logo]<http://www.cloudmoyo.com/?utm_source=signature&utm_medium=mail%20sign&utm_campaign=mail%20sign>
[https://icertisportalstorage.blob.core.windows.net/siteasset/icon-linkedin.png]<https://www.linkedin.com/company/cloudmoyo>[https://icertisportalstorage.blob.core.windows.net/siteasset/icon-fb.png]<https://www.facebook.com/gocloudmoyo>[https://icertisportalstorage.blob.core.windows.net/siteasset/icon-twitter.png]<https://twitter.com/GoCloudMoyo>
www.cloudmoyo.com
IMPORTANT NOTICE: This communication, including any attachment, contains information that may be confidential or privileged, and is intended solely for the entity or individual to whom it is addressed. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message is strictly prohibited. Nothing in this email, including any attachment, is intended to be a legally binding signature.