MapReduceIndexerTool erroring with max_array_length

[Darshan Pandya <da...@gmail.com>]

Hello,

While this may be a question for cloudera, I wanted to tap the brains of
this very active community as well.

I am trying to use the MapReduceIndexerTool to index data in a hive table
to Solr Cloud / Cloudera Search.

The tool is failing the job with the following error



1799 [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  - Indexing 1
files using 1 real mappers into 10 reducers

Error: MAX_ARRAY_LENGTH

Error: MAX_ARRAY_LENGTH

Error: MAX_ARRAY_LENGTH

36962 [main] ERROR org.apache.solr.hadoop.MapReduceIndexerTool  - Job
failed! jobName: org.apache.solr.hadoop.MapReduceIndexerTool/MorphlineMapper,
jobId: job_1473161870114_0339



The error stack trace is

2016-09-08 10:39:20,128 ERROR [main]
org.apache.hadoop.mapred.YarnChild: Error running child :
java.lang.NoSuchFieldError: MAX_ARRAY_LENGTH
	at org.apache.lucene.codecs.memory.DirectDocValuesFormat.<clinit>(DirectDocValuesFormat.java:58)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at java.lang.Class.newInstance(Class.java:374)
	at org.apache.lucene.util.NamedSPILoader.reload(NamedSPILoader.java:67)
	at org.apache.lucene.util.NamedSPILoader.<init>(NamedSPILoader.java:47)
	at org.apache.lucene.util.NamedSPILoader.<init>(NamedSPILoader.java:37)
	at org.apache.lucene.codecs.DocValuesFormat.<clinit>(DocValuesFormat.java:43)
	at org.apache.solr.core.SolrResourceLoader.reloadLuceneSPI(SolrResourceLoader.java:205)





My Schema.xml looks like



<fields>

   <field name="dataset_id" type="string" indexed="true" stored="true"
required="true" multiValued="false" docValue="true" />

   <field name="search_string" type="string" indexed="true" stored="true"
docValue="true"/>

   <field name="_version_" type="long" indexed="true" stored="true"/>

</fields>





<!-- Field to use to determine and enforce document uniqueness.

      Unless this field is marked with required="false", it will be a
required field

   -->

<uniqueKey>dataset_id</uniqueKey>





I am otherwise about to post documents using Solr APIs / upload methods.
Only the MapReduceIndexer tool is failing.



The command I am using is

hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar
org.apache.solr.hadoop.MapReduceIndexerTool -D
'mapred.child.java.opts=-Xmx500m' --log4j
/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/s
hare/doc/search-1.0.0+cdh5.7.0+0/examples/solr-nrt/log4j.properties
--morphline-file /home/$USER/morphline2.conf --output-dir
hdfs://NNHOST:8020/user/$USER/outdir --verbose --zk-host ZKHOST:2181/solr1
--collection dataCatalog_search_index
hdfs://NNHOST:8020/user/hive/warehouse/name.db/concatenated_index4/;



My morphline config looks like



SOLR_LOCATOR : {

  # Name of solr collection

  collection : search_index



  # ZooKeeper ensemble

  $zkHost:2181/solr1"

}



# Specify an array of one or more morphlines, each of which defines an ETL

# transformation chain. A morphline consists of one or more (potentially

# nested) commands. A morphline is a way to consume records (e.g. Flume
events,

# HDFS files or blocks), turn them into a stream of records, and pipe the
stream

# of records through a set of easily configurable transformations on the
way to

# a target application such as Solr.

morphlines : [

  {

    id : search_index

    importCommands : ["org.kitesdk.**", "org.apache.solr.**"]

    commands : [

      {

        readCSV {

          separator : ","

          columns : [dataset_id,search_string]

          ignoreFirstLine : true

          charset : UTF-8

        }

      }





      # Consume the output record of the previous command and pipe another

      # record downstream.

      #

      # Command that deletes record fields that are unknown to Solr

      # schema.xml.

      #

      # Recall that Solr throws an exception on any attempt to load a
document

      # that contains a field that isn't specified in schema.xml.

      {

        sanitizeUnknownSolrFields {

          # Location from which to fetch Solr schema

          solrLocator : ${SOLR_LOCATOR}

        }

      }



      # log the record at DEBUG level to SLF4J

      { logDebug { format : "output record: {}", args : ["@{}"] } }



      # load the record into a Solr server or MapReduce Reducer

      {

        loadSolr {

          solrLocator : ${SOLR_LOCATOR}

        }

      }

    ]

  }

]





Please let me know if I am going anything wrong.

-- 
Sincerely,
Darshan