You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Prasanth M Sasidharan <pr...@gmail.com> on 2022/05/27 11:00:29 UTC
Hive unable to Launch job to spark
Hello team,
I am trying to use spark as the engine for Hive.
Hive version : 3.1.1
Spark: 2.4.0 (have tried with several versions)
I am able to query the hive tables from within pyspark.However, when i
execute the hive query from the hive shell, it throws the following error
Launch Command: "/opt/CDS/pre_requisites/jdk1.8.0_141/bin/java" "-cp"
"/opt/CDS/spark/conf/:/opt/CDS/spark/jars/*:/opt/CDS/hadoop-2.9.2/etc/hadoop/:/etc/tez/conf:/opt/CDS/hive/conf/:/opt/CDS/hive/lib/HikariCP-2.6.1.jar:/opt/CDS/hive/lib/ST4-4.0.4.jar:/opt/CDS/hive/lib/accumulo-core-1.7.3.jar:/opt/CDS/hive/lib/accumulo-fate-1.7.3.jar:/opt/CDS/hive/lib/accumulo-start-1.7.3.jar:/opt/CDS/hive/lib/accumulo-trace-1.7.3.jar:/opt/CDS/hive/lib/aircompressor-0.10.jar:/opt/CDS/hive/lib/ant-1.9.1.jar:/opt/CDS/hive/lib/ant-launcher-1.9.1.jar:/opt/CDS/hive/lib/antlr-runtime-3.5.2.jar:/opt/CDS/hive/lib/antlr4-runtime-4.5.jar:/opt/CDS/hive/lib/aopalliance-repackaged-2.5.0-b32.jar:/opt/CDS/hive/lib/apache-jsp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/apache-jstl-9.3.20.v20170531.jar:/opt/CDS/hive/lib/arrow-format-0.8.0.jar:/opt/CDS/hive/lib/arrow-memory-0.8.0.jar:/opt/CDS/hive/lib/arrow-vector-0.8.0.jar:/opt/CDS/hive/lib/asm-5.0.1.jar:/opt/CDS/hive/lib/asm-commons-5.0.1.jar:/opt/CDS/hive/lib/asm-tree-5.0.1.jar:/opt/CDS/hive/lib/audience-annotations-0.5.0.jar:/opt/CDS/hive/lib/avatica-1.11.0.jar:/opt/CDS/hive/lib/avro-1.7.7.jar:/opt/CDS/hive/lib/bonecp-0.8.0.RELEASE.jar:/opt/CDS/hive/lib/calcite-core-1.16.0.jar:/opt/CDS/hive/lib/calcite-druid-1.16.0.jar:/opt/CDS/hive/lib/calcite-linq4j-1.16.0.jar:/opt/CDS/hive/lib/commons-cli-1.2.jar:/opt/CDS/hive/lib/commons-codec-1.7.jar:/opt/CDS/hive/lib/commons-collections4-4.1.jar:/opt/CDS/hive/lib/commons-compiler-2.7.6.jar:/opt/CDS/hive/lib/commons-compress-1.9.jar:/opt/CDS/hive/lib/commons-crypto-1.0.0.jar:/opt/CDS/hive/lib/commons-dbcp-1.4.jar:/opt/CDS/hive/lib/commons-io-2.4.jar:/opt/CDS/hive/lib/commons-lang-2.6.jar:/opt/CDS/hive/lib/commons-lang3-3.2.jar:/opt/CDS/hive/lib/commons-logging-1.0.4.jar:/opt/CDS/hive/lib/commons-math-2.1.jar:/opt/CDS/hive/lib/commons-math3-3.6.1.jar:/opt/CDS/hive/lib/commons-pool-1.5.4.jar:/opt/CDS/hive/lib/commons-vfs2-2.1.jar:/opt/CDS/hive/lib/curator-client-2.12.0.jar:/opt/CDS/hive/lib/curator-framework-2.12.0.jar:/opt/CDS/hive/lib/curator-recipes-2.12.0.jar:/opt/CDS/hive/lib/datanucleus-api-jdo-4.2.4.jar:/opt/CDS/hive/lib/datanucleus-core-4.1.17.jar:/opt/CDS/hive/lib/datanucleus-rdbms-4.1.19.jar:/opt/CDS/hive/lib/derby-10.14.1.0.jar:/opt/CDS/hive/lib/disruptor-3.3.6.jar:/opt/CDS/hive/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/CDS/hive/lib/druid-hdfs-storage-0.12.0.jar:/opt/CDS/hive/lib/ecj-4.4.2.jar:/opt/CDS/hive/lib/esri-geometry-api-2.0.0.jar:/opt/CDS/hive/lib/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hive/lib/flatbuffers-1.2.0-3f79e055.jar:/opt/CDS/hive/lib/groovy-all-2.4.11.jar:/opt/CDS/hive/lib/gson-2.2.4.jar:/opt/CDS/hive/lib/guava-19.0.jar:/opt/CDS/hive/lib/hbase-client-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-http-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-mapreduce-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-api-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-prefix-tree-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-procedure-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-shaded-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-replication-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-server-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-shaded-miscellaneous-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-netty-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-protobuf-1.0.1.jar:/opt/CDS/hive/lib/hive-accumulo-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-beeline-3.1.1.jar:/opt/CDS/hive/lib/hive-classification-3.1.1.jar:/opt/CDS/hive/lib/hive-cli-3.1.1.jar:/opt/CDS/hive/lib/hive-common-3.1.1.jar:/opt/CDS/hive/lib/hive-contrib-3.1.1.jar:/opt/CDS/hive/lib/hive-druid-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-exec-3.1.1.jar:/opt/CDS/hive/lib/hive-hbase-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-core-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-server-extensions-3.1.1.jar:/opt/CDS/hive/lib/hive-hplsql-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-kryo-registrator-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1-tests.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-ext-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-server-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-tez-3.1.1.jar:/opt/CDS/hive/lib/hive-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-serde-3.1.1.jar:/opt/CDS/hive/lib/hive-service-3.1.1.jar:/opt/CDS/hive/lib/hive-service-rpc-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-0.23-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-common-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-scheduler-3.1.1.jar:/opt/CDS/hive/lib/hive-standalone-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-storage-api-2.7.0.jar:/opt/CDS/hive/lib/hive-streaming-3.1.1.jar:/opt/CDS/hive/lib/hive-testutils-3.1.1.jar:/opt/CDS/hive/lib/hive-upgrade-acid-3.1.1.jar:/opt/CDS/hive/lib/hive-vector-code-gen-3.1.1.jar:/opt/CDS/hive/lib/hk2-api-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-locator-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-utils-2.5.0-b32.jar:/opt/CDS/hive/lib/hppc-0.7.2.jar:/opt/CDS/hive/lib/htrace-core-3.2.0-incubating.jar:/opt/CDS/hive/lib/httpclient-4.5.2.jar:/opt/CDS/hive/lib/httpcore-4.4.4.jar:/opt/CDS/hive/lib/ivy-2.4.0.jar:/opt/CDS/hive/lib/jackson-annotations-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-asl-1.9.13.jar:/opt/CDS/hive/lib/jackson-databind-2.9.5.jar:/opt/CDS/hive/lib/jackson-dataformat-smile-2.9.5.jar:/opt/CDS/hive/lib/jackson-mapper-asl-1.9.13.jar:/opt/CDS/hive/lib/jamon-runtime-2.3.1.jar:/opt/CDS/hive/lib/janino-2.7.6.jar:/opt/CDS/hive/lib/javassist-3.20.0-GA.jar:/opt/CDS/hive/lib/javax.annotation-api-1.2.jar:/opt/CDS/hive/lib/javax.inject-2.5.0-b32.jar:/opt/CDS/hive/lib/javax.jdo-3.2.0-m3.jar:/opt/CDS/hive/lib/javax.servlet-api-3.1.0.jar:/opt/CDS/hive/lib/javax.servlet.jsp-2.3.2.jar:/opt/CDS/hive/lib/javax.servlet.jsp-api-2.3.1.jar:/opt/CDS/hive/lib/javax.ws.rs-api-2.0.1.jar:/opt/CDS/hive/lib/javolution-5.5.1.jar:/opt/CDS/hive/lib/jcodings-1.0.18.jar:/opt/CDS/hive/lib/jcommander-1.32.jar:/opt/CDS/hive/lib/jdo-api-3.0.1.jar:/opt/CDS/hive/lib/jersey-client-2.25.1.jar:/opt/CDS/hive/lib/jersey-common-2.25.1.jar:/opt/CDS/hive/lib/jersey-container-servlet-core-2.25.1.jar:/opt/CDS/hive/lib/jersey-guava-2.25.1.jar:/opt/CDS/hive/lib/jersey-media-jaxb-2.25.1.jar:/opt/CDS/hive/lib/jersey-server-2.25.1.jar:/opt/CDS/hive/lib/jettison-1.1.jar:/opt/CDS/hive/lib/jetty-annotations-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-http-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-io-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jaas-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jndi-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-plus-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-rewrite-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-runner-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-schemas-3.1.jar:/opt/CDS/hive/lib/jetty-security-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-util-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-webapp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-xml-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jline-2.12.jar:/opt/CDS/hive/lib/joda-time-2.9.9.jar:/opt/CDS/hive/lib/joni-2.1.11.jar:/opt/CDS/hive/lib/jpam-1.1.jar:/opt/CDS/hive/lib/json-1.8.jar:/opt/CDS/hive/lib/jsr305-3.0.0.jar:/opt/CDS/hive/lib/jta-1.1.jar:/opt/CDS/hive/lib/kryo-shaded-4.0.2.jar:/opt/CDS/hive/lib/libfb303-0.9.3.jar:/opt/CDS/hive/lib/libthrift-0.9.3.jar:/opt/CDS/hive/lib/memory-0.9.0.jar:/opt/CDS/hive/lib/metrics-core-3.1.0.jar:/opt/CDS/hive/lib/metrics-json-3.1.0.jar:/opt/CDS/hive/lib/metrics-jvm-3.1.0.jar:/opt/CDS/hive/lib/minlog-1.3.0.jar:/opt/CDS/hive/lib/mysql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/netty-3.10.5.Final.jar:/opt/CDS/hive/lib/netty-all-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-buffer-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-common-4.1.17.Final.jar:/opt/CDS/hive/lib/opencsv-2.3.jar:/opt/CDS/hive/lib/orc-core-1.5.1.jar:/opt/CDS/hive/lib/orc-shims-1.5.1.jar:/opt/CDS/hive/lib/org.abego.treelayout.core-1.0.1.jar:/opt/CDS/hive/lib/osgi-resource-locator-1.0.1.jar:/opt/CDS/hive/lib/paranamer-2.3.jar:/opt/CDS/hive/lib/parquet-hadoop-bundle-1.10.0.jar:/opt/CDS/hive/lib/postgresql-9.4.1208.jre7.jar:/opt/CDS/hive/lib/postgresql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/protobuf-java-2.5.0.jar:/opt/CDS/hive/lib/scala-library-2.11.12.jar:/opt/CDS/hive/lib/sketches-core-0.9.0.jar:/opt/CDS/hive/lib/snappy-java-1.1.4.jar:/opt/CDS/hive/lib/spark-core_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-network-common_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-unsafe_2.11-2.4.0.jar:/opt/CDS/hive/lib/sqlline-1.3.0.jar:/opt/CDS/hive/lib/stax-api-1.0.1.jar:/opt/CDS/hive/lib/super-csv-2.2.0.jar:/opt/CDS/hive/lib/taglibs-standard-impl-1.2.5.jar:/opt/CDS/hive/lib/taglibs-standard-spec-1.2.5.jar:/opt/CDS/hive/lib/tempus-fugit-1.1.jar:/opt/CDS/hive/lib/transaction-api-1.1.jar:/opt/CDS/hive/lib/validation-api-1.1.0.Final.jar:/opt/CDS/hive/lib/velocity-1.5.jar:/opt/CDS/hive/lib/websocket-api-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-common-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/zookeeper-3.4.6.jar:/opt/CDS/hadoop-2.9.2/share/hadoop/tools/lib/hadoop-distcp-2.9.2.jar:/opt/CDS/hbase//conf/:/opt/CDS/hbase//lib/shaded-clients/hbase-shaded-mapreduce-2.1.1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/commons-logging-1.2.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/log4j-1.2.17.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/opt/CDS/hive/lib/log4j-1.2-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-core-2.10.0.jar:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar:/opt/CDS/hive/lib/log4j-web-2.10.0.jar:/opt/CDS/hadoop-2.9.2/contrib/capacity-scheduler/*.jar:/opt/CDS/hadoop-2.9.2//share/hadoop/common/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/common/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/lib/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/*"
"-Xmx4096M"
"-Dspark.hadoop.hbase.http.filter.initializers=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter"
"-Dspark.hadoop.hbase.defaults.for.version=2.0.0-alpha4"
"-Dspark.hadoop.hbase.wal.dir.perms=700"
"-Dspark.hadoop.hbase.hstore.checksum.algorithm=CRC32C"
"-Dspark.hadoop.hbase.regionserver.thrift.framed.max_frame_size_in_mb=2"
"-Dspark.hadoop.hbase.snapshot.restore.take.failsafe.snapshot=true"
"-Dspark.hadoop.hbase.regionserver.port=60020"
"-Dspark.hadoop.hbase.rpc.rows.warning.threshold=5000"
"-Dspark.hadoop.hbase.storescanner.parallel.seek.enable=false"
"-Dspark.hadoop.hbase.client.max.perregion.tasks=1"
"-Dspark.hadoop.hbase.mob.compaction.threads.max=1"
"-Dspark.hadoop.hbase.mob.compaction.mergeable.threshold=1342177280"
"-Dspark.hadoop.hbase.rpc.timeout=60000"
"-Dspark.hadoop.hbase.replication.rpc.codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags"
"-Dspark.serializer=org.apache.spark.serializer.KryoSerializer"
"-Dspark.hadoop.hbase.auth.token.max.lifetime=604800000"
"-Dspark.hadoop.hbase.mob.compaction.chore.period=604800"
"-Dspark.hadoop.hbase.bulkload.retries.number=10"
"-Dspark.hadoop.hbase.hregion.memstore.flush.size=134217728"
"-Dspark.master=spark://172.30.61.226:7077"
"-Dspark.hadoop.hbase.master.port=60000"
"-Dspark.hadoop.hbase.server.versionfile.writeattempts=3"
"-Dspark.hadoop.hbase.regionserver.logroll.errors.tolerated=2"
"-Dspark.hadoop.hbase.unsafe.stream.capability.enforce=false"
"-Dspark.hadoop.hbase.master.info.port=60010" "-Dspark.app.name=Hive on
Spark (sessionId = 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)"
"-Dspark.hadoop.hbase.table.lock.enable=true"
"-Dspark.hadoop.hbase.lease.recovery.dfs.timeout=64000"
"-Dspark.hadoop.hbase.regionserver.thrift.compact=false"
"-Dspark.hadoop.hbase.hregion.majorcompaction=604800000"
"-Dspark.hadoop.hbase.zookeeper.peerport=2888"
"-Dspark.hadoop.hbase.cluster.distributed=true"
"-Dspark.hadoop.hbase.regionserver.storefile.refresh.period=0"
"-Dspark.kryo.classesToRegister=org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,org.apache.hadoop.io.Writable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.io.HiveKey"
"-Dspark.hadoop.hbase.regionserver.optionalcacheflushinterval=3600000"
"-Dspark.hadoop.hbase.master.procedurewalcleaner.ttl=604800000"
"-Dspark.hadoop.hbase.zookeeper.quorum=172.30.61.229,172.30.61.230,172.30.61.231"
"-Dspark.sql.warehouse.dir=/hive/cdswarehouse"
"-Dspark.hadoop.fs.defaultFS=hdfs://cdscluster/"
"-Dspark.hadoop.hbase.hstore.blockingStoreFiles=10"
"-Dspark.hadoop.hbase.regionserver.hlog.reader.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader"
"-Dspark.hadoop.hbase.regionserver.checksum.verify=true"
"-Dspark.hadoop.hbase.rest.readonly=false"
"-Dspark.hadoop.hbase.regionserver.majorcompaction.pagecache.drop=true"
"-Dspark.submit.deployMode=cluster"
"-Dspark.hadoop.hbase.http.max.threads=16"
"-Dspark.hadoop.hbase.data.umask.enable=false"
"-Dspark.hadoop.hbase.hstore.flusher.count=2"
"-Dspark.hadoop.hbase.server.compactchecker.interval.multiplier=1000"
"-Dspark.hadoop.hbase.hstore.compaction.max.size=9223372036854775807"
"-Dspark.hadoop.hbase.rest.filter.classes=org.apache.hadoop.hbase.rest.filter.GzipFilter"
"-Dspark.hadoop.hbase.rest.csrf.enabled=false"
"-Dspark.hadoop.hbase.coordinated.state.manager.class=org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager"
"-Dspark.hadoop.hbase.mob.delfile.max.count=3"
"-Dspark.hadoop.hbase.master.loadbalancer.class=org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer"
"-Dspark.hadoop.hbase.snapshot.region.timeout=300000"
"-Dspark.hadoop.hbase.hstore.compactionThreshold=3"
"-Dspark.hadoop.hbase.data.umask=000"
"-Dspark.client.authentication.secret=0ffffffea690fffffffb4d0ffffffc045240ffffffff410ffffffd56874181d04700ffffffae320ffffffc40ffffffe036c1a321c63e5e0ffffffde213a3d"
"-Dspark.hadoop.hbase.hstore.bytes.per.checksum=16384"
"-Dspark.hadoop.hbase.server.scanner.max.result.size=104857600"
"-Dspark.hadoop.hbase.ipc.server.callqueue.scan.ratio=0"
"-Dspark.hadoop.hbase.ipc.client.tcpnodelay=true"
"-Dspark.hadoop.hbase.hstore.compaction.ratio.offpeak=5.0F"
"-Dspark.hadoop.hbase.zookeeper.property.dataDir=/data/CDS/zookeeper"
"-Dspark.jars=file:/opt/CDS/hive/lib/hive-exec-3.1.1.jar"
"-Dspark.hadoop.hbase.client.pause=100"
"-Dspark.hadoop.hbase.coprocessor.enabled=true"
"-Dspark.hadoop.hbase.thrift.minWorkerThreads=16"
"-Dspark.kryo.referenceTracking=false"
"-Dspark.hadoop.hbase.master.mob.ttl.cleaner.period=86400"
"-Dspark.hadoop.hbase.status.published=false"
"-Dspark.hadoop.hbase.client.retries.number=35"
"-Dspark.hadoop.hbase.hstore.blockingWaitTime=90000"
"-Dspark.hadoop.hbase.mob.cache.evict.remain.ratio=0.5f"
"-Dspark.hadoop.hbase.master.normalizer.class=org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer"
"-Dspark.hadoop.hbase.client.scanner.caching=2147483647"
"-Dspark.hadoop.hbase.normalizer.period=300000"
"-Dspark.hadoop.hbase.hregion.max.filesize=10737418240"
"-Dspark.hadoop.hbase.hregion.memstore.mslab.enabled=true"
"-Dspark.hadoop.hbase.hstore.compaction.ratio=1.2F"
"-Dspark.hadoop.hbase.security.visibility.mutations.checkauths=false"
"-Dspark.hadoop.hbase.client.localityCheck.threadPoolSize=2"
"-Dspark.hadoop.hbase.master.hfilecleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner"
"-Dspark.hadoop.hbase.coprocessor.user.enabled=true"
"-Dspark.hadoop.hbase.rs.cacheblocksonwrite=false"
"-Dspark.executor.memory=6048m"
"-Dspark.hadoop.hbase.regionserver.msginterval=3000"
"-Dspark.hadoop.hbase.snapshot.master.timeout.millis=300000"
"-Dspark.executor.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/"
"-Dspark.hadoop.cloneConf=true"
"-Dspark.hadoop.hbase.fs.tmp.dir=/user/cdsuser/hbase-staging"
"-Dspark.hadoop.hbase.mob.cache.evict.period=3600"
"-Dspark.hadoop.hbase.security.exec.permission.checks=false"
"-Dspark.hadoop.hbase.hregion.percolumnfamilyflush.size.lower.bound.min=16777216"
"-Dspark.hadoop.hbase.zookeeper.property.syncLimit=5"
"-Dspark.history.fs.logDirectory=hdfs://cdscluster/spark-logs"
"-Dspark.hadoop.hbase.client.operation.timeout=1200000"
"-Dspark.hadoop.hbase.ipc.server.callqueue.handler.factor=0.1"
"-Dspark.hadoop.hbase.rootdir=hdfs://cdscluster/hbase"
"-Dspark.hadoop.hbase.ipc.server.callqueue.read.ratio=0"
"-Dspark.eventLog.enabled=true"
"-Dspark.hadoop.hbase.regionserver.handler.abort.on.error.percent=0.5"
"-Dspark.hadoop.hbase.thrift.maxWorkerThreads=1000"
"-Dspark.hadoop.hbase.regionserver.logroll.period=3600000"
"-Dspark.hadoop.hbase.mob.file.cache.size=1000"
"-Dspark.hadoop.hbase.master.logcleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner"
"-Dspark.hadoop.hbase.coprocessor.abortonerror=true"
"-Dspark.hadoop.hbase.hregion.preclose.flush.size=5242880"
"-Dspark.hadoop.hbase.dynamic.jars.dir=hdfs://cdscluster/hbase/lib"
"-Dspark.hadoop.hbase.column.max.version=1"
"-Dspark.hadoop.hbase.master.logcleaner.ttl=600000"
"-Dspark.hadoop.hbase.regionserver.hostname.disable.master.reversedns=false"
"-Dspark.hadoop.hbase.replication.source.maxthreads=10"
"-Dspark.hadoop.hbase.client.scanner.timeout.period=60000"
"-Dspark.hadoop.hbase.security.authentication=simple"
"-Dspark.hadoop.hbase.snapshot.enabled=true"
"-Dspark.hadoop.hbase.master.balancer.maxRitPercent=1.0"
"-Dspark.hadoop.hbase.zookeeper.dns.interface=default"
"-Dspark.driver.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/"
"-Dspark.hadoop.hbase.display.keys=true"
"-Dspark.hadoop.hbase.regionserver.info.port.auto=false"
"-Dspark.hadoop.hbase.client.write.buffer=2097152"
"-Dspark.hadoop.hbase.regionserver.minorcompaction.pagecache.drop=true"
"-Dspark.hadoop.hbase.server.keyvalue.maxsize=10485760"
"-Dspark.history.fs.update.interval=10s"
"-Dspark.hadoop.hbase.rest.threads.max=100"
"-Dspark.hadoop.hbase.systemtables.compacting.memstore.type=NONE"
"-Dspark.hadoop.hbase.client.keyvalue.maxsize=10485760"
"-Dspark.hadoop.hbase.client.max.total.tasks=100"
"-Dspark.hadoop.hbase.defaults.for.version.skip=false"
"-Dspark.hadoop.hbase.zookeeper.property.initLimit=10"
"-Dspark.hadoop.hbase.mob.compaction.batch.size=100"
"-Dspark.hadoop.hbase.master.info.bindAddress=0.0.0.0"
"-Dspark.hadoop.hbase.regionserver.handler.count=30"
"-Dspark.hadoop.hbase.regionserver.region.split.policy=org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy"
"-Dspark.hadoop.hbase.rpc.shortoperation.timeout=10000"
"-Dspark.hadoop.hbase.zookeeper.property.clientPort=2181"
"-Dspark.hadoop.hbase.master.infoserver.redirect=true"
"-Dspark.hadoop.hbase.hstore.compaction.min.size=134217728"
"-Dspark.hadoop.hbase.hstore.compaction.max=10"
"-Dspark.hadoop.hbase.thrift.maxQueuedRequests=1000"
"-Dspark.driver.supervise=false" "-Dspark.sql.catalogImplementation=hive"
"-Dspark.hadoop.hbase.local.dir=/tmp/hbase-cdsuser/local/"
"-Dspark.hadoop.hbase.ipc.client.fallback-to-simple-auth-allowed=false"
"-Dspark.hadoop.hbase.hstore.compaction.kv.max=10"
"-Dspark.hadoop.hbase.regionserver.info.bindAddress=0.0.0.0"
"-Dspark.hadoop.hbase.hregion.memstore.block.multiplier=4"
"-Dspark.hadoop.hbase.regionserver.regionSplitLimit=1000"
"-Dspark.hadoop.hbase.bucketcache.combinedcache.enabled=true"
"-Dspark.eventLog.dir=/tmp"
"-Dspark.hadoop.hbase.regionserver.dns.interface=default"
"-Dspark.hadoop.hbase.storescanner.parallel.seek.threads=10"
"-Dspark.hadoop.hbase.rest.port=8080"
"-Dspark.hadoop.hbase.http.staticuser.user=dr.stack"
"-Dspark.hadoop.hbase.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*"
"-Dspark.client.authentication.client_id=9a1af6d4-59d9-49fc-b8ab-40469737cc9c"
"-Dspark.hadoop.hbase.ipc.server.fallback-to-simple-auth-allowed=false"
"-Dspark.hadoop.hbase.regionserver.hlog.writer.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter"
"-Dspark.driver.memory=4g"
"-Dspark.hadoop.hbase.client.perserver.requests.threshold=2147483647"
"-Dspark.hadoop.hbase.regionserver.dns.nameserver=default"
"-Dspark.hadoop.hbase.client.scanner.max.result.size=2097152"
"-Dspark.hadoop.hbase.auth.key.update.interval=86400000"
"-Dspark.hadoop.hbase.balancer.period=300000" "-
Dspark.hadoop.hbase.snapshot.restore.failsafe.name=hbase-failsafe-{
snapshot.name}-{restore.timestamp}"
"-Dspark.hadoop.hbase.status.publisher.class=org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher"
"-Dspark.hadoop.hbase.rest.support.proxyuser=false"
"-Dspark.hadoop.hbase.regionserver.thrift.framed=false"
"-Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider"
"-Dspark.hadoop.hbase.status.multicast.address.port=16100"
"-Dspark.hadoop.hbase.master.hostname=nl-itc-l-cdma00002.labs.liberty.biz"
"-Dspark.rpc.askTimeout=10s"
"-Dspark.hadoop.hbase.client.max.perserver.tasks=2"
"-Dspark.hadoop.hbase.serial.replication.waitingMs=10000"
"-Dspark.hadoop.hbase.mob.compactor.class=org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactor"
"-Dspark.history.ui.port=18080"
"-Dspark.hadoop.hbase.zookeeper.property.maxClientCnxns=300"
"-Dspark.hadoop.hbase.regions.slop=0.001"
"-Dspark.hadoop.hbase.tmp.dir=/tmp/hbase-cdsuser"
"-Dspark.hadoop.hbase.offpeak.end.hour=-1"
"-Dspark.hadoop.zookeeper.znode.acl.parent=acl"
"-Dspark.hadoop.hbase.server.thread.wakefrequency=10000"
"-Dspark.hadoop.hbase.hregion.majorcompaction.jitter=0.50"
"-Dspark.hadoop.hbase.region.replica.replication.enabled=false"
"-Dspark.hadoop.hbase.zookeeper.leaderport=3888"
"-Dspark.hadoop.hbase.regionserver.thread.compaction.throttle=2684354560"
"-Dspark.hadoop.hbase.offpeak.start.hour=-1"
"-Dspark.hadoop.hbase.rest.threads.min=2"
"-Dspark.hadoop.hbase.regionserver.info.port=60030"
"-Dspark.hadoop.hbase.lease.recovery.timeout=900000"
"-Dspark.executor.cores=6" "-Dspark.hadoop.hbase.rootdir.perms=700"
"-Dspark.hadoop.hbase.status.listener.class=org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener"
"-Dspark.hadoop.hbase.hstore.time.to.purge.deletes=0"
"-Dspark.hadoop.hbase.zookeeper.dns.nameserver=default"
"-Dspark.hadoop.hbase.table.max.rowsize=1073741824"
"-Dspark.hadoop.hbase.hstore.compaction.min=3"
"-Dspark.hadoop.hbase.dfs.client.read.shortcircuit.buffer.size=131072"
"-Dspark.hadoop.hbase.cells.scanned.per.heartbeat.check=10000"
"-Dspark.hadoop.zookeeper.znode.parent=/hbase"
"-Dspark.hadoop.hbase.status.multicast.address.ip=226.1.1.3"
"org.apache.spark.deploy.worker.DriverWrapper" "spark://
Worker@172.30.61.226:41266"
"/opt/CDS/spark/work/driver-20220527123634-0000/hive-exec-3.1.1.jar"
"org.apache.hive.spark.client.RemoteDriver" "--remote-host" "
nl-itc-l-cdma00001.labs.liberty.biz" "--remote-port" "36667" "--conf"
"hive.spark.client.connect.timeout=30000" "--conf"
"hive.spark.client.server.connect.timeout=300000" "--conf"
"hive.spark.client.channel.log.level=null" "--conf"
"hive.spark.client.rpc.max.size=52428800" "--conf"
"hive.spark.client.rpc.threads=8" "--conf"
"hive.spark.client.secret.bits=256" "--conf"
"hive.spark.client.rpc.server.address=172.30.61.226"
========================================
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/CDS/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/CDS/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
at
org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:48)
at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:138)
at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:536)
... 6 more
and eventually timesout
following is the content of *spark-defaults.conf*
*######################################################################################################################*
spark.master spark://172.30.61.226:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://cdscluster/spark/
spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory hdfs://cdscluster/spark-logs
spark.history.fs.update.interval 10s
spark.history.ui.port 18080
spark.sql.catalogImplementation=hive
spark.sql.warehouse.dir /hive/cdswarehouse
spark.serializer org.apache.spark.serializer.KryoSerializer
*######################################################################################################################*
*hive-site.xml*
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/hive/cdswarehouse</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby://
172.30.61.100:1527/cds_hive_metastoredb;create=true</value>
</property>
<property>
<name>datanucleus.schema.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.apache.derby.jdbc.ClientDriver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
<property>
<name>hive.execution.engine</name>
<value>spark</value>
</property>
<property>
<name>spark.master</name>
<value>spark://172.30.61.226:7077</value>
</property>
<property>
<name>hive.metastore.port</name>
<value>9083</value>
</property>
<property>
<name>mapreduce.input.fileinputformat.input.dir.recursive</name>
<value>true</value>
</property>
<property>
<name>spark.eventLog.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.spark.client.rpc.server.address</name>
<value>172.30.61.226</value>
</property>
<property>
<name>spark.eventLog.dir</name>
<value>/tmp</value>
</property>
<property>
<name>spark.executor.memory</name>
<value>6048m</value>
</property>
<property>
<name>spark.executor.cores</name>
<value>6</value>
</property>
<property>
<name>hive.spark.client.connect.timeout</name>
<value>30000ms</value>
</property>
<property>
<name>hive.spark.client.server.connect.timeout</name>
<value>300000ms</value>
</property>
</configuration>
*######################################################################################################################*
Have followed the instructions in [
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started]
and
have linked the jars
lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35
spark-core_2.11-2.4.0.jar -> /opt/CDS/spark/jars/spark-core_2.11-2.4.0.jar
lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35
scala-library-2.11.12.jar -> /opt/CDS/spark/jars/scala-library-2.11.12.jar
lrwxrwxrwx 1 cdsuser cdsuser 55 May 27 12:35
spark-network-common_2.11-2.4.0.jar ->
/opt/CDS/spark/jars/spark-network-common_2.11-2.4.0.jar
lrwxrwxrwx 1 cdsuser cdsuser 47 May 27 12:35
spark-unsafe_2.11-2.4.0.jar ->
/opt/CDS/spark/jars/spark-unsafe_2.11-2.4.0.jar
*######################################################################################################################*
*Hive console*
*##################################*
hive> select count(*) from cds.spectrum_data;
2022-05-27 12:36:31,955 INFO [e32ede4e-2110-4aea-9068-40e4a4986f78 main]
reducesink.VectorReduceSinkEmptyKeyOperator:
VectorReduceSinkEmptyKeyOperator constructor vectorReduceSinkInfo
org.apache.hadoop.hive.ql.plan.VectorReduceSinkInfo@62dfe152
Query ID = cdsuser_20220527123629_e7e2aa6a-0223-4943-b387-a24f09e26aa8
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Failed to execute spark task, with exception
'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark
client for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)'
FAILED: Execution Error, return code 30041 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark
client for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c
############################################################################################################
Would really appreciate it if you can guide me on this.
--
Live every day as if it were your last, because one of these days, it will
be.
Regards,
Prasanth M Sasidharan
Re: Hive unable to Launch job to spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Agreed. The last time I presented it was in 2016
Presentation in London: Query Engines for Hive: MR, Spark, Tez with LLAP
– Considerations!
<https://www.slideshare.net/MichTalebzadeh1/query-engines-for-hive-mr-spark-tez-with-llap-considerations>
and sadly they diverged!
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Tue, 31 May 2022 at 10:29, Peter Vary <pv...@cloudera.com> wrote:
> Hi Prasanth,
>
> I would suggest not to invest too heavily in Hive on Spark.
> I recent years there was no movement around the feature and it will be
> removed in Hive 4.0.0.
>
> Thanks,
> Peter
>
> On 2022. May 27., at 13:00, Prasanth M Sasidharan <pr...@gmail.com>
> wrote:
>
> Hello team,
>
> I am trying to use spark as the engine for Hive.
> Hive version : 3.1.1
> Spark: 2.4.0 (have tried with several versions)
>
> I am able to query the hive tables from within pyspark.However, when i
> execute the hive query from the hive shell, it throws the following error
>
> Launch Command: "/opt/CDS/pre_requisites/jdk1.8.0_141/bin/java" "-cp"
> "/opt/CDS/spark/conf/:/opt/CDS/spark/jars/*:/opt/CDS/hadoop-2.9.2/etc/hadoop/:/etc/tez/conf:/opt/CDS/hive/conf/:/opt/CDS/hive/lib/HikariCP-2.6.1.jar:/opt/CDS/hive/lib/ST4-4.0.4.jar:/opt/CDS/hive/lib/accumulo-core-1.7.3.jar:/opt/CDS/hive/lib/accumulo-fate-1.7.3.jar:/opt/CDS/hive/lib/accumulo-start-1.7.3.jar:/opt/CDS/hive/lib/accumulo-trace-1.7.3.jar:/opt/CDS/hive/lib/aircompressor-0.10.jar:/opt/CDS/hive/lib/ant-1.9.1.jar:/opt/CDS/hive/lib/ant-launcher-1.9.1.jar:/opt/CDS/hive/lib/antlr-runtime-3.5.2.jar:/opt/CDS/hive/lib/antlr4-runtime-4.5.jar:/opt/CDS/hive/lib/aopalliance-repackaged-2.5.0-b32.jar:/opt/CDS/hive/lib/apache-jsp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/apache-jstl-9.3.20.v20170531.jar:/opt/CDS/hive/lib/arrow-format-0.8.0.jar:/opt/CDS/hive/lib/arrow-memory-0.8.0.jar:/opt/CDS/hive/lib/arrow-vector-0.8.0.jar:/opt/CDS/hive/lib/asm-5.0.1.jar:/opt/CDS/hive/lib/asm-commons-5.0.1.jar:/opt/CDS/hive/lib/asm-tree-5.0.1.jar:/opt/CDS/hive/lib/audience-annotations-0.5.0.jar:/opt/CDS/hive/lib/avatica-1.11.0.jar:/opt/CDS/hive/lib/avro-1.7.7.jar:/opt/CDS/hive/lib/bonecp-0.8.0.RELEASE.jar:/opt/CDS/hive/lib/calcite-core-1.16.0.jar:/opt/CDS/hive/lib/calcite-druid-1.16.0.jar:/opt/CDS/hive/lib/calcite-linq4j-1.16.0.jar:/opt/CDS/hive/lib/commons-cli-1.2.jar:/opt/CDS/hive/lib/commons-codec-1.7.jar:/opt/CDS/hive/lib/commons-collections4-4.1.jar:/opt/CDS/hive/lib/commons-compiler-2.7.6.jar:/opt/CDS/hive/lib/commons-compress-1.9.jar:/opt/CDS/hive/lib/commons-crypto-1.0.0.jar:/opt/CDS/hive/lib/commons-dbcp-1.4.jar:/opt/CDS/hive/lib/commons-io-2.4.jar:/opt/CDS/hive/lib/commons-lang-2.6.jar:/opt/CDS/hive/lib/commons-lang3-3.2.jar:/opt/CDS/hive/lib/commons-logging-1.0.4.jar:/opt/CDS/hive/lib/commons-math-2.1.jar:/opt/CDS/hive/lib/commons-math3-3.6.1.jar:/opt/CDS/hive/lib/commons-pool-1.5.4.jar:/opt/CDS/hive/lib/commons-vfs2-2.1.jar:/opt/CDS/hive/lib/curator-client-2.12.0.jar:/opt/CDS/hive/lib/curator-framework-2.12.0.jar:/opt/CDS/hive/lib/curator-recipes-2.12.0.jar:/opt/CDS/hive/lib/datanucleus-api-jdo-4.2.4.jar:/opt/CDS/hive/lib/datanucleus-core-4.1.17.jar:/opt/CDS/hive/lib/datanucleus-rdbms-4.1.19.jar:/opt/CDS/hive/lib/derby-10.14.1.0.jar:/opt/CDS/hive/lib/disruptor-3.3.6.jar:/opt/CDS/hive/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/CDS/hive/lib/druid-hdfs-storage-0.12.0.jar:/opt/CDS/hive/lib/ecj-4.4.2.jar:/opt/CDS/hive/lib/esri-geometry-api-2.0.0.jar:/opt/CDS/hive/lib/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hive/lib/flatbuffers-1.2.0-3f79e055.jar:/opt/CDS/hive/lib/groovy-all-2.4.11.jar:/opt/CDS/hive/lib/gson-2.2.4.jar:/opt/CDS/hive/lib/guava-19.0.jar:/opt/CDS/hive/lib/hbase-client-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-http-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-mapreduce-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-api-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-prefix-tree-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-procedure-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-shaded-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-replication-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-server-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-shaded-miscellaneous-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-netty-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-protobuf-1.0.1.jar:/opt/CDS/hive/lib/hive-accumulo-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-beeline-3.1.1.jar:/opt/CDS/hive/lib/hive-classification-3.1.1.jar:/opt/CDS/hive/lib/hive-cli-3.1.1.jar:/opt/CDS/hive/lib/hive-common-3.1.1.jar:/opt/CDS/hive/lib/hive-contrib-3.1.1.jar:/opt/CDS/hive/lib/hive-druid-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-exec-3.1.1.jar:/opt/CDS/hive/lib/hive-hbase-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-core-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-server-extensions-3.1.1.jar:/opt/CDS/hive/lib/hive-hplsql-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-kryo-registrator-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1-tests.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-ext-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-server-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-tez-3.1.1.jar:/opt/CDS/hive/lib/hive-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-serde-3.1.1.jar:/opt/CDS/hive/lib/hive-service-3.1.1.jar:/opt/CDS/hive/lib/hive-service-rpc-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-0.23-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-common-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-scheduler-3.1.1.jar:/opt/CDS/hive/lib/hive-standalone-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-storage-api-2.7.0.jar:/opt/CDS/hive/lib/hive-streaming-3.1.1.jar:/opt/CDS/hive/lib/hive-testutils-3.1.1.jar:/opt/CDS/hive/lib/hive-upgrade-acid-3.1.1.jar:/opt/CDS/hive/lib/hive-vector-code-gen-3.1.1.jar:/opt/CDS/hive/lib/hk2-api-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-locator-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-utils-2.5.0-b32.jar:/opt/CDS/hive/lib/hppc-0.7.2.jar:/opt/CDS/hive/lib/htrace-core-3.2.0-incubating.jar:/opt/CDS/hive/lib/httpclient-4.5.2.jar:/opt/CDS/hive/lib/httpcore-4.4.4.jar:/opt/CDS/hive/lib/ivy-2.4.0.jar:/opt/CDS/hive/lib/jackson-annotations-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-asl-1.9.13.jar:/opt/CDS/hive/lib/jackson-databind-2.9.5.jar:/opt/CDS/hive/lib/jackson-dataformat-smile-2.9.5.jar:/opt/CDS/hive/lib/jackson-mapper-asl-1.9.13.jar:/opt/CDS/hive/lib/jamon-runtime-2.3.1.jar:/opt/CDS/hive/lib/janino-2.7.6.jar:/opt/CDS/hive/lib/javassist-3.20.0-GA.jar:/opt/CDS/hive/lib/javax.annotation-api-1.2.jar:/opt/CDS/hive/lib/javax.inject-2.5.0-b32.jar:/opt/CDS/hive/lib/javax.jdo-3.2.0-m3.jar:/opt/CDS/hive/lib/javax.servlet-api-3.1.0.jar:/opt/CDS/hive/lib/javax.servlet.jsp-2.3.2.jar:/opt/CDS/hive/lib/javax.servlet.jsp-api-2.3.1.jar:/opt/CDS/hive/lib/
> javax.ws.rs-api-2.0.1.jar:/opt/CDS/hive/lib/javolution-5.5.1.jar:/opt/CDS/hive/lib/jcodings-1.0.18.jar:/opt/CDS/hive/lib/jcommander-1.32.jar:/opt/CDS/hive/lib/jdo-api-3.0.1.jar:/opt/CDS/hive/lib/jersey-client-2.25.1.jar:/opt/CDS/hive/lib/jersey-common-2.25.1.jar:/opt/CDS/hive/lib/jersey-container-servlet-core-2.25.1.jar:/opt/CDS/hive/lib/jersey-guava-2.25.1.jar:/opt/CDS/hive/lib/jersey-media-jaxb-2.25.1.jar:/opt/CDS/hive/lib/jersey-server-2.25.1.jar:/opt/CDS/hive/lib/jettison-1.1.jar:/opt/CDS/hive/lib/jetty-annotations-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-http-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-io-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jaas-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jndi-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-plus-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-rewrite-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-runner-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-schemas-3.1.jar:/opt/CDS/hive/lib/jetty-security-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-util-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-webapp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-xml-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jline-2.12.jar:/opt/CDS/hive/lib/joda-time-2.9.9.jar:/opt/CDS/hive/lib/joni-2.1.11.jar:/opt/CDS/hive/lib/jpam-1.1.jar:/opt/CDS/hive/lib/json-1.8.jar:/opt/CDS/hive/lib/jsr305-3.0.0.jar:/opt/CDS/hive/lib/jta-1.1.jar:/opt/CDS/hive/lib/kryo-shaded-4.0.2.jar:/opt/CDS/hive/lib/libfb303-0.9.3.jar:/opt/CDS/hive/lib/libthrift-0.9.3.jar:/opt/CDS/hive/lib/memory-0.9.0.jar:/opt/CDS/hive/lib/metrics-core-3.1.0.jar:/opt/CDS/hive/lib/metrics-json-3.1.0.jar:/opt/CDS/hive/lib/metrics-jvm-3.1.0.jar:/opt/CDS/hive/lib/minlog-1.3.0.jar:/opt/CDS/hive/lib/mysql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/netty-3.10.5.Final.jar:/opt/CDS/hive/lib/netty-all-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-buffer-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-common-4.1.17.Final.jar:/opt/CDS/hive/lib/opencsv-2.3.jar:/opt/CDS/hive/lib/orc-core-1.5.1.jar:/opt/CDS/hive/lib/orc-shims-1.5.1.jar:/opt/CDS/hive/lib/org.abego.treelayout.core-1.0.1.jar:/opt/CDS/hive/lib/osgi-resource-locator-1.0.1.jar:/opt/CDS/hive/lib/paranamer-2.3.jar:/opt/CDS/hive/lib/parquet-hadoop-bundle-1.10.0.jar:/opt/CDS/hive/lib/postgresql-9.4.1208.jre7.jar:/opt/CDS/hive/lib/postgresql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/protobuf-java-2.5.0.jar:/opt/CDS/hive/lib/scala-library-2.11.12.jar:/opt/CDS/hive/lib/sketches-core-0.9.0.jar:/opt/CDS/hive/lib/snappy-java-1.1.4.jar:/opt/CDS/hive/lib/spark-core_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-network-common_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-unsafe_2.11-2.4.0.jar:/opt/CDS/hive/lib/sqlline-1.3.0.jar:/opt/CDS/hive/lib/stax-api-1.0.1.jar:/opt/CDS/hive/lib/super-csv-2.2.0.jar:/opt/CDS/hive/lib/taglibs-standard-impl-1.2.5.jar:/opt/CDS/hive/lib/taglibs-standard-spec-1.2.5.jar:/opt/CDS/hive/lib/tempus-fugit-1.1.jar:/opt/CDS/hive/lib/transaction-api-1.1.jar:/opt/CDS/hive/lib/validation-api-1.1.0.Final.jar:/opt/CDS/hive/lib/velocity-1.5.jar:/opt/CDS/hive/lib/websocket-api-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-common-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/zookeeper-3.4.6.jar:/opt/CDS/hadoop-2.9.2/share/hadoop/tools/lib/hadoop-distcp-2.9.2.jar:/opt/CDS/hbase//conf/:/opt/CDS/hbase//lib/shaded-clients/hbase-shaded-mapreduce-2.1.1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/commons-logging-1.2.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/log4j-1.2.17.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/opt/CDS/hive/lib/log4j-1.2-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-core-2.10.0.jar:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar:/opt/CDS/hive/lib/log4j-web-2.10.0.jar:/opt/CDS/hadoop-2.9.2/contrib/capacity-scheduler/*.jar:/opt/CDS/hadoop-2.9.2//share/hadoop/common/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/common/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/lib/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/*"
> "-Xmx4096M"
> "-Dspark.hadoop.hbase.http.filter.initializers=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter"
> "-Dspark.hadoop.hbase.defaults.for.version=2.0.0-alpha4"
> "-Dspark.hadoop.hbase.wal.dir.perms=700"
> "-Dspark.hadoop.hbase.hstore.checksum.algorithm=CRC32C"
> "-Dspark.hadoop.hbase.regionserver.thrift.framed.max_frame_size_in_mb=2"
> "-Dspark.hadoop.hbase.snapshot.restore.take.failsafe.snapshot=true"
> "-Dspark.hadoop.hbase.regionserver.port=60020"
> "-Dspark.hadoop.hbase.rpc.rows.warning.threshold=5000"
> "-Dspark.hadoop.hbase.storescanner.parallel.seek.enable=false"
> "-Dspark.hadoop.hbase.client.max.perregion.tasks=1"
> "-Dspark.hadoop.hbase.mob.compaction.threads.max=1"
> "-Dspark.hadoop.hbase.mob.compaction.mergeable.threshold=1342177280"
> "-Dspark.hadoop.hbase.rpc.timeout=60000"
> "-Dspark.hadoop.hbase.replication.rpc.codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags"
> "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer"
> "-Dspark.hadoop.hbase.auth.token.max.lifetime=604800000"
> "-Dspark.hadoop.hbase.mob.compaction.chore.period=604800"
> "-Dspark.hadoop.hbase.bulkload.retries.number=10"
> "-Dspark.hadoop.hbase.hregion.memstore.flush.size=134217728"
> "-Dspark.master=spark://172.30.61.226:7077"
> "-Dspark.hadoop.hbase.master.port=60000"
> "-Dspark.hadoop.hbase.server.versionfile.writeattempts=3"
> "-Dspark.hadoop.hbase.regionserver.logroll.errors.tolerated=2"
> "-Dspark.hadoop.hbase.unsafe.stream.capability.enforce=false"
> "-Dspark.hadoop.hbase.master.info.port=60010" "-Dspark.app.name
> <http://dspark.app.name/>=Hive on Spark (sessionId =
> 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)"
> "-Dspark.hadoop.hbase.table.lock.enable=true"
> "-Dspark.hadoop.hbase.lease.recovery.dfs.timeout=64000"
> "-Dspark.hadoop.hbase.regionserver.thrift.compact=false"
> "-Dspark.hadoop.hbase.hregion.majorcompaction=604800000"
> "-Dspark.hadoop.hbase.zookeeper.peerport=2888"
> "-Dspark.hadoop.hbase.cluster.distributed=true"
> "-Dspark.hadoop.hbase.regionserver.storefile.refresh.period=0"
> "-Dspark.kryo.classesToRegister=org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,org.apache.hadoop.io.Writable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.io.HiveKey"
> "-Dspark.hadoop.hbase.regionserver.optionalcacheflushinterval=3600000"
> "-Dspark.hadoop.hbase.master.procedurewalcleaner.ttl=604800000"
> "-Dspark.hadoop.hbase.zookeeper.quorum=172.30.61.229,172.30.61.230,172.30.61.231"
> "-Dspark.sql.warehouse.dir=/hive/cdswarehouse" "-Dspark.hadoop.fs.defaultFS=
> hdfs://cdscluster/" "-Dspark.hadoop.hbase.hstore.blockingStoreFiles=10"
> "-Dspark.hadoop.hbase.regionserver.hlog.reader.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader"
> "-Dspark.hadoop.hbase.regionserver.checksum.verify=true"
> "-Dspark.hadoop.hbase.rest.readonly=false"
> "-Dspark.hadoop.hbase.regionserver.majorcompaction.pagecache.drop=true"
> "-Dspark.submit.deployMode=cluster"
> "-Dspark.hadoop.hbase.http.max.threads=16"
> "-Dspark.hadoop.hbase.data.umask.enable=false"
> "-Dspark.hadoop.hbase.hstore.flusher.count=2"
> "-Dspark.hadoop.hbase.server.compactchecker.interval.multiplier=1000"
> "-Dspark.hadoop.hbase.hstore.compaction.max.size=9223372036854775807"
> "-Dspark.hadoop.hbase.rest.filter.classes=org.apache.hadoop.hbase.rest.filter.GzipFilter"
> "-Dspark.hadoop.hbase.rest.csrf.enabled=false"
> "-Dspark.hadoop.hbase.coordinated.state.manager.class=org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager"
> "-Dspark.hadoop.hbase.mob.delfile.max.count=3"
> "-Dspark.hadoop.hbase.master.loadbalancer.class=org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer"
> "-Dspark.hadoop.hbase.snapshot.region.timeout=300000"
> "-Dspark.hadoop.hbase.hstore.compactionThreshold=3"
> "-Dspark.hadoop.hbase.data.umask=000"
> "-Dspark.client.authentication.secret=0ffffffea690fffffffb4d0ffffffc045240ffffffff410ffffffd56874181d04700ffffffae320ffffffc40ffffffe036c1a321c63e5e0ffffffde213a3d"
> "-Dspark.hadoop.hbase.hstore.bytes.per.checksum=16384"
> "-Dspark.hadoop.hbase.server.scanner.max.result.size=104857600"
> "-Dspark.hadoop.hbase.ipc.server.callqueue.scan.ratio=0"
> "-Dspark.hadoop.hbase.ipc.client.tcpnodelay=true"
> "-Dspark.hadoop.hbase.hstore.compaction.ratio.offpeak=5.0F"
> "-Dspark.hadoop.hbase.zookeeper.property.dataDir=/data/CDS/zookeeper"
> "-Dspark.jars=file:/opt/CDS/hive/lib/hive-exec-3.1.1.jar"
> "-Dspark.hadoop.hbase.client.pause=100"
> "-Dspark.hadoop.hbase.coprocessor.enabled=true"
> "-Dspark.hadoop.hbase.thrift.minWorkerThreads=16"
> "-Dspark.kryo.referenceTracking=false"
> "-Dspark.hadoop.hbase.master.mob.ttl.cleaner.period=86400"
> "-Dspark.hadoop.hbase.status.published=false"
> "-Dspark.hadoop.hbase.client.retries.number=35"
> "-Dspark.hadoop.hbase.hstore.blockingWaitTime=90000"
> "-Dspark.hadoop.hbase.mob.cache.evict.remain.ratio=0.5f"
> "-Dspark.hadoop.hbase.master.normalizer.class=org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer"
> "-Dspark.hadoop.hbase.client.scanner.caching=2147483647"
> "-Dspark.hadoop.hbase.normalizer.period=300000"
> "-Dspark.hadoop.hbase.hregion.max.filesize=10737418240"
> "-Dspark.hadoop.hbase.hregion.memstore.mslab.enabled=true"
> "-Dspark.hadoop.hbase.hstore.compaction.ratio=1.2F"
> "-Dspark.hadoop.hbase.security.visibility.mutations.checkauths=false"
> "-Dspark.hadoop.hbase.client.localityCheck.threadPoolSize=2"
> "-Dspark.hadoop.hbase.master.hfilecleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner"
> "-Dspark.hadoop.hbase.coprocessor.user.enabled=true"
> "-Dspark.hadoop.hbase.rs.cacheblocksonwrite=false"
> "-Dspark.executor.memory=6048m"
> "-Dspark.hadoop.hbase.regionserver.msginterval=3000"
> "-Dspark.hadoop.hbase.snapshot.master.timeout.millis=300000"
> "-Dspark.executor.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/"
> "-Dspark.hadoop.cloneConf=true"
> "-Dspark.hadoop.hbase.fs.tmp.dir=/user/cdsuser/hbase-staging"
> "-Dspark.hadoop.hbase.mob.cache.evict.period=3600"
> "-Dspark.hadoop.hbase.security.exec.permission.checks=false"
> "-Dspark.hadoop.hbase.hregion.percolumnfamilyflush.size.lower.bound.min=16777216"
> "-Dspark.hadoop.hbase.zookeeper.property.syncLimit=5"
> "-Dspark.history.fs.logDirectory=hdfs://cdscluster/spark-logs"
> "-Dspark.hadoop.hbase.client.operation.timeout=1200000"
> "-Dspark.hadoop.hbase.ipc.server.callqueue.handler.factor=0.1"
> "-Dspark.hadoop.hbase.rootdir=hdfs://cdscluster/hbase"
> "-Dspark.hadoop.hbase.ipc.server.callqueue.read.ratio=0"
> "-Dspark.eventLog.enabled=true"
> "-Dspark.hadoop.hbase.regionserver.handler.abort.on.error.percent=0.5"
> "-Dspark.hadoop.hbase.thrift.maxWorkerThreads=1000"
> "-Dspark.hadoop.hbase.regionserver.logroll.period=3600000"
> "-Dspark.hadoop.hbase.mob.file.cache.size=1000"
> "-Dspark.hadoop.hbase.master.logcleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner"
> "-Dspark.hadoop.hbase.coprocessor.abortonerror=true"
> "-Dspark.hadoop.hbase.hregion.preclose.flush.size=5242880"
> "-Dspark.hadoop.hbase.dynamic.jars.dir=hdfs://cdscluster/hbase/lib"
> "-Dspark.hadoop.hbase.column.max.version=1"
> "-Dspark.hadoop.hbase.master.logcleaner.ttl=600000"
> "-Dspark.hadoop.hbase.regionserver.hostname.disable.master.reversedns=false"
> "-Dspark.hadoop.hbase.replication.source.maxthreads=10"
> "-Dspark.hadoop.hbase.client.scanner.timeout.period=60000"
> "-Dspark.hadoop.hbase.security.authentication=simple"
> "-Dspark.hadoop.hbase.snapshot.enabled=true"
> "-Dspark.hadoop.hbase.master.balancer.maxRitPercent=1.0"
> "-Dspark.hadoop.hbase.zookeeper.dns.interface=default"
> "-Dspark.driver.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/"
> "-Dspark.hadoop.hbase.display.keys=true"
> "-Dspark.hadoop.hbase.regionserver.info.port.auto=false"
> "-Dspark.hadoop.hbase.client.write.buffer=2097152"
> "-Dspark.hadoop.hbase.regionserver.minorcompaction.pagecache.drop=true"
> "-Dspark.hadoop.hbase.server.keyvalue.maxsize=10485760"
> "-Dspark.history.fs.update.interval=10s"
> "-Dspark.hadoop.hbase.rest.threads.max=100"
> "-Dspark.hadoop.hbase.systemtables.compacting.memstore.type=NONE"
> "-Dspark.hadoop.hbase.client.keyvalue.maxsize=10485760"
> "-Dspark.hadoop.hbase.client.max.total.tasks=100"
> "-Dspark.hadoop.hbase.defaults.for.version.skip=false"
> "-Dspark.hadoop.hbase.zookeeper.property.initLimit=10"
> "-Dspark.hadoop.hbase.mob.compaction.batch.size=100"
> "-Dspark.hadoop.hbase.master.info.bindAddress=0.0.0.0"
> "-Dspark.hadoop.hbase.regionserver.handler.count=30"
> "-Dspark.hadoop.hbase.regionserver.region.split.policy=org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy"
> "-Dspark.hadoop.hbase.rpc.shortoperation.timeout=10000"
> "-Dspark.hadoop.hbase.zookeeper.property.clientPort=2181"
> "-Dspark.hadoop.hbase.master.infoserver.redirect=true"
> "-Dspark.hadoop.hbase.hstore.compaction.min.size=134217728"
> "-Dspark.hadoop.hbase.hstore.compaction.max=10"
> "-Dspark.hadoop.hbase.thrift.maxQueuedRequests=1000"
> "-Dspark.driver.supervise=false" "-Dspark.sql.catalogImplementation=hive"
> "-Dspark.hadoop.hbase.local.dir=/tmp/hbase-cdsuser/local/"
> "-Dspark.hadoop.hbase.ipc.client.fallback-to-simple-auth-allowed=false"
> "-Dspark.hadoop.hbase.hstore.compaction.kv.max=10"
> "-Dspark.hadoop.hbase.regionserver.info.bindAddress=0.0.0.0"
> "-Dspark.hadoop.hbase.hregion.memstore.block.multiplier=4"
> "-Dspark.hadoop.hbase.regionserver.regionSplitLimit=1000"
> "-Dspark.hadoop.hbase.bucketcache.combinedcache.enabled=true"
> "-Dspark.eventLog.dir=/tmp"
> "-Dspark.hadoop.hbase.regionserver.dns.interface=default"
> "-Dspark.hadoop.hbase.storescanner.parallel.seek.threads=10"
> "-Dspark.hadoop.hbase.rest.port=8080"
> "-Dspark.hadoop.hbase.http.staticuser.user=dr.stack"
> "-Dspark.hadoop.hbase.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*"
> "-Dspark.client.authentication.client_id=9a1af6d4-59d9-49fc-b8ab-40469737cc9c"
> "-Dspark.hadoop.hbase.ipc.server.fallback-to-simple-auth-allowed=false"
> "-Dspark.hadoop.hbase.regionserver.hlog.writer.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter"
> "-Dspark.driver.memory=4g"
> "-Dspark.hadoop.hbase.client.perserver.requests.threshold=2147483647"
> "-Dspark.hadoop.hbase.regionserver.dns.nameserver=default"
> "-Dspark.hadoop.hbase.client.scanner.max.result.size=2097152"
> "-Dspark.hadoop.hbase.auth.key.update.interval=86400000"
> "-Dspark.hadoop.hbase.balancer.period=300000" "-
> Dspark.hadoop.hbase.snapshot.restore.failsafe.name
> <http://dspark.hadoop.hbase.snapshot.restore.failsafe.name/>
> =hbase-failsafe-{snapshot.name}-{restore.timestamp}"
> "-Dspark.hadoop.hbase.status.publisher.class=org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher"
> "-Dspark.hadoop.hbase.rest.support.proxyuser=false"
> "-Dspark.hadoop.hbase.regionserver.thrift.framed=false"
> "-Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider"
> "-Dspark.hadoop.hbase.status.multicast.address.port=16100"
> "-Dspark.hadoop.hbase.master.hostname=nl-itc-l-cdma00002.labs.liberty.biz"
> "-Dspark.rpc.askTimeout=10s"
> "-Dspark.hadoop.hbase.client.max.perserver.tasks=2"
> "-Dspark.hadoop.hbase.serial.replication.waitingMs=10000"
> "-Dspark.hadoop.hbase.mob.compactor.class=org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactor"
> "-Dspark.history.ui.port=18080"
> "-Dspark.hadoop.hbase.zookeeper.property.maxClientCnxns=300"
> "-Dspark.hadoop.hbase.regions.slop=0.001"
> "-Dspark.hadoop.hbase.tmp.dir=/tmp/hbase-cdsuser"
> "-Dspark.hadoop.hbase.offpeak.end.hour=-1"
> "-Dspark.hadoop.zookeeper.znode.acl.parent=acl"
> "-Dspark.hadoop.hbase.server.thread.wakefrequency=10000"
> "-Dspark.hadoop.hbase.hregion.majorcompaction.jitter=0.50"
> "-Dspark.hadoop.hbase.region.replica.replication.enabled=false"
> "-Dspark.hadoop.hbase.zookeeper.leaderport=3888"
> "-Dspark.hadoop.hbase.regionserver.thread.compaction.throttle=2684354560"
> "-Dspark.hadoop.hbase.offpeak.start.hour=-1"
> "-Dspark.hadoop.hbase.rest.threads.min=2"
> "-Dspark.hadoop.hbase.regionserver.info.port=60030"
> "-Dspark.hadoop.hbase.lease.recovery.timeout=900000"
> "-Dspark.executor.cores=6" "-Dspark.hadoop.hbase.rootdir.perms=700"
> "-Dspark.hadoop.hbase.status.listener.class=org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener"
> "-Dspark.hadoop.hbase.hstore.time.to.purge.deletes=0"
> "-Dspark.hadoop.hbase.zookeeper.dns.nameserver=default"
> "-Dspark.hadoop.hbase.table.max.rowsize=1073741824"
> "-Dspark.hadoop.hbase.hstore.compaction.min=3"
> "-Dspark.hadoop.hbase.dfs.client.read.shortcircuit.buffer.size=131072"
> "-Dspark.hadoop.hbase.cells.scanned.per.heartbeat.check=10000"
> "-Dspark.hadoop.zookeeper.znode.parent=/hbase"
> "-Dspark.hadoop.hbase.status.multicast.address.ip=226.1.1.3"
> "org.apache.spark.deploy.worker.DriverWrapper" "spark://
> Worker@172.30.61.226:41266"
> "/opt/CDS/spark/work/driver-20220527123634-0000/hive-exec-3.1.1.jar"
> "org.apache.hive.spark.client.RemoteDriver" "--remote-host" "
> nl-itc-l-cdma00001.labs.liberty.biz" "--remote-port" "36667" "--conf"
> "hive.spark.client.connect.timeout=30000" "--conf"
> "hive.spark.client.server.connect.timeout=300000" "--conf"
> "hive.spark.client.channel.log.level=null" "--conf"
> "hive.spark.client.rpc.max.size=52428800" "--conf"
> "hive.spark.client.rpc.threads=8" "--conf"
> "hive.spark.client.secret.bits=256" "--conf"
> "hive.spark.client.rpc.server.address=172.30.61.226"
> ========================================
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/CDS/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/CDS/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
> at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> Caused by: java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
> at
> org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:48)
> at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:138)
> at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:536)
> ... 6 more
>
> and eventually timesout
>
> following is the content of *spark-defaults.conf*
>
> *######################################################################################################################*
> spark.master spark://172.30.61.226:7077
> spark.eventLog.enabled true
> spark.eventLog.dir hdfs://cdscluster/spark/
> spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
> spark.history.fs.logDirectory hdfs://cdscluster/spark-logs
> spark.history.fs.update.interval 10s
> spark.history.ui.port 18080
> spark.sql.catalogImplementation=hive
> spark.sql.warehouse.dir /hive/cdswarehouse
> spark.serializer
> org.apache.spark.serializer.KryoSerializer
>
> *######################################################################################################################*
> *hive-site.xml*
> <?xml version="1.0" encoding="UTF-8" standalone="no"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <configuration>
>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>/hive/cdswarehouse</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:derby://
> 172.30.61.100:1527/cds_hive_metastoredb;create=true</value>
> </property>
> <property>
> <name>datanucleus.schema.autoCreateTables</name>
> <value>true</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>org.apache.derby.jdbc.ClientDriver</value>
> <description>Driver class name for a JDBC metastore</description>
> </property>
> <property>
> <name>hive.server2.thrift.port</name>
> <value>10000</value>
> </property>
>
> <property>
> <name>hive.server2.enable.doAs</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.execution.engine</name>
> <value>spark</value>
> </property>
> <property>
> <name>spark.master</name>
> <value>spark://172.30.61.226:7077</value>
> </property>
> <property>
> <name>hive.metastore.port</name>
> <value>9083</value>
> </property>
>
> <property>
> <name>mapreduce.input.fileinputformat.input.dir.recursive</name>
> <value>true</value>
> </property>
> <property>
> <name>spark.eventLog.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.spark.client.rpc.server.address</name>
> <value>172.30.61.226</value>
> </property>
>
> <property>
> <name>spark.eventLog.dir</name>
> <value>/tmp</value>
> </property>
> <property>
> <name>spark.executor.memory</name>
> <value>6048m</value>
> </property>
> <property>
> <name>spark.executor.cores</name>
> <value>6</value>
> </property>
>
> <property>
> <name>hive.spark.client.connect.timeout</name>
> <value>30000ms</value>
> </property>
> <property>
> <name>hive.spark.client.server.connect.timeout</name>
> <value>300000ms</value>
> </property>
> </configuration>
>
> *######################################################################################################################*
> Have followed the instructions in [
> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started] and
> have linked the jars
>
> lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35
> spark-core_2.11-2.4.0.jar -> /opt/CDS/spark/jars/spark-core_2.11-2.4.0.jar
> lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35
> scala-library-2.11.12.jar -> /opt/CDS/spark/jars/scala-library-2.11.12.jar
> lrwxrwxrwx 1 cdsuser cdsuser 55 May 27 12:35
> spark-network-common_2.11-2.4.0.jar ->
> /opt/CDS/spark/jars/spark-network-common_2.11-2.4.0.jar
> lrwxrwxrwx 1 cdsuser cdsuser 47 May 27 12:35
> spark-unsafe_2.11-2.4.0.jar ->
> /opt/CDS/spark/jars/spark-unsafe_2.11-2.4.0.jar
>
> *######################################################################################################################*
>
> *Hive console*
> *##################################*
> hive> select count(*) from cds.spectrum_data;
> 2022-05-27 12:36:31,955 INFO [e32ede4e-2110-4aea-9068-40e4a4986f78 main]
> reducesink.VectorReduceSinkEmptyKeyOperator:
> VectorReduceSinkEmptyKeyOperator constructor vectorReduceSinkInfo
> org.apache.hadoop.hive.ql.plan.VectorReduceSinkInfo@62dfe152
> Query ID = cdsuser_20220527123629_e7e2aa6a-0223-4943-b387-a24f09e26aa8
> Total jobs = 1
> Launching Job 1 out of 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=<number>
> Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark
> client for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)'
> FAILED: Execution Error, return code 30041 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark
> client for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c
>
> ############################################################################################################
> Would really appreciate it if you can guide me on this.
>
>
> --
> Live every day as if it were your last, because one of these days, it will
> be.
>
> Regards,
>
> Prasanth M Sasidharan
>
>
>
Re: Hive unable to Launch job to spark
Posted by Peter Vary <pv...@cloudera.com>.
Hi Prasanth,
I would suggest not to invest too heavily in Hive on Spark.
I recent years there was no movement around the feature and it will be removed in Hive 4.0.0.
Thanks,
Peter
> On 2022. May 27., at 13:00, Prasanth M Sasidharan <pr...@gmail.com> wrote:
>
> Hello team,
>
> I am trying to use spark as the engine for Hive.
> Hive version : 3.1.1
> Spark: 2.4.0 (have tried with several versions)
>
> I am able to query the hive tables from within pyspark.However, when i execute the hive query from the hive shell, it throws the following error
>
> Launch Command: "/opt/CDS/pre_requisites/jdk1.8.0_141/bin/java" "-cp" "/opt/CDS/spark/conf/:/opt/CDS/spark/jars/*:/opt/CDS/hadoop-2.9.2/etc/hadoop/:/etc/tez/conf:/opt/CDS/hive/conf/:/opt/CDS/hive/lib/HikariCP-2.6.1.jar:/opt/CDS/hive/lib/ST4-4.0.4.jar:/opt/CDS/hive/lib/accumulo-core-1.7.3.jar:/opt/CDS/hive/lib/accumulo-fate-1.7.3.jar:/opt/CDS/hive/lib/accumulo-start-1.7.3.jar:/opt/CDS/hive/lib/accumulo-trace-1.7.3.jar:/opt/CDS/hive/lib/aircompressor-0.10.jar:/opt/CDS/hive/lib/ant-1.9.1.jar:/opt/CDS/hive/lib/ant-launcher-1.9.1.jar:/opt/CDS/hive/lib/antlr-runtime-3.5.2.jar:/opt/CDS/hive/lib/antlr4-runtime-4.5.jar:/opt/CDS/hive/lib/aopalliance-repackaged-2.5.0-b32.jar:/opt/CDS/hive/lib/apache-jsp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/apache-jstl-9.3.20.v20170531.jar:/opt/CDS/hive/lib/arrow-format-0.8.0.jar:/opt/CDS/hive/lib/arrow-memory-0.8.0.jar:/opt/CDS/hive/lib/arrow-vector-0.8.0.jar:/opt/CDS/hive/lib/asm-5.0.1.jar:/opt/CDS/hive/lib/asm-commons-5.0.1.jar:/opt/CDS/hive/lib/asm-tree-5.0.1.jar:/opt/CDS/hive/lib/audience-annotations-0.5.0.jar:/opt/CDS/hive/lib/avatica-1.11.0.jar:/opt/CDS/hive/lib/avro-1.7.7.jar:/opt/CDS/hive/lib/bonecp-0.8.0.RELEASE.jar:/opt/CDS/hive/lib/calcite-core-1.16.0.jar:/opt/CDS/hive/lib/calcite-druid-1.16.0.jar:/opt/CDS/hive/lib/calcite-linq4j-1.16.0.jar:/opt/CDS/hive/lib/commons-cli-1.2.jar:/opt/CDS/hive/lib/commons-codec-1.7.jar:/opt/CDS/hive/lib/commons-collections4-4.1.jar:/opt/CDS/hive/lib/commons-compiler-2.7.6.jar:/opt/CDS/hive/lib/commons-compress-1.9.jar:/opt/CDS/hive/lib/commons-crypto-1.0.0.jar:/opt/CDS/hive/lib/commons-dbcp-1.4.jar:/opt/CDS/hive/lib/commons-io-2.4.jar:/opt/CDS/hive/lib/commons-lang-2.6.jar:/opt/CDS/hive/lib/commons-lang3-3.2.jar:/opt/CDS/hive/lib/commons-logging-1.0.4.jar:/opt/CDS/hive/lib/commons-math-2.1.jar:/opt/CDS/hive/lib/commons-math3-3.6.1.jar:/opt/CDS/hive/lib/commons-pool-1.5.4.jar:/opt/CDS/hive/lib/commons-vfs2-2.1.jar:/opt/CDS/hive/lib/curator-client-2.12.0.jar:/opt/CDS/hive/lib/curator-framework-2.12.0.jar:/opt/CDS/hive/lib/curator-recipes-2.12.0.jar:/opt/CDS/hive/lib/datanucleus-api-jdo-4.2.4.jar:/opt/CDS/hive/lib/datanucleus-core-4.1.17.jar:/opt/CDS/hive/lib/datanucleus-rdbms-4.1.19.jar:/opt/CDS/hive/lib/derby-10.14.1.0.jar:/opt/CDS/hive/lib/disruptor-3.3.6.jar:/opt/CDS/hive/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/CDS/hive/lib/druid-hdfs-storage-0.12.0.jar:/opt/CDS/hive/lib/ecj-4.4.2.jar:/opt/CDS/hive/lib/esri-geometry-api-2.0.0.jar:/opt/CDS/hive/lib/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hive/lib/flatbuffers-1.2.0-3f79e055.jar:/opt/CDS/hive/lib/groovy-all-2.4.11.jar:/opt/CDS/hive/lib/gson-2.2.4.jar:/opt/CDS/hive/lib/guava-19.0.jar:/opt/CDS/hive/lib/hbase-client-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-common-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4-tests.jar:/opt/CDS/hive/lib/hbase-hadoop2-compat-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-http-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-mapreduce-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-metrics-api-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-prefix-tree-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-procedure-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-protocol-shaded-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-replication-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-server-2.0.0-alpha4.jar:/opt/CDS/hive/lib/hbase-shaded-miscellaneous-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-netty-1.0.1.jar:/opt/CDS/hive/lib/hbase-shaded-protobuf-1.0.1.jar:/opt/CDS/hive/lib/hive-accumulo-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-beeline-3.1.1.jar:/opt/CDS/hive/lib/hive-classification-3.1.1.jar:/opt/CDS/hive/lib/hive-cli-3.1.1.jar:/opt/CDS/hive/lib/hive-common-3.1.1.jar:/opt/CDS/hive/lib/hive-contrib-3.1.1.jar:/opt/CDS/hive/lib/hive-druid-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-exec-3.1.1.jar:/opt/CDS/hive/lib/hive-hbase-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-core-3.1.1.jar:/opt/CDS/hive/lib/hive-hcatalog-server-extensions-3.1.1.jar:/opt/CDS/hive/lib/hive-hplsql-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-3.1.1.jar:/opt/CDS/hive/lib/hive-jdbc-handler-3.1.1.jar:/opt/CDS/hive/lib/hive-kryo-registrator-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1-tests.jar:/opt/CDS/hive/lib/hive-llap-common-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-ext-client-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-server-3.1.1.jar:/opt/CDS/hive/lib/hive-llap-tez-3.1.1.jar:/opt/CDS/hive/lib/hive-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-serde-3.1.1.jar:/opt/CDS/hive/lib/hive-service-3.1.1.jar:/opt/CDS/hive/lib/hive-service-rpc-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-0.23-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-common-3.1.1.jar:/opt/CDS/hive/lib/hive-shims-scheduler-3.1.1.jar:/opt/CDS/hive/lib/hive-standalone-metastore-3.1.1.jar:/opt/CDS/hive/lib/hive-storage-api-2.7.0.jar:/opt/CDS/hive/lib/hive-streaming-3.1.1.jar:/opt/CDS/hive/lib/hive-testutils-3.1.1.jar:/opt/CDS/hive/lib/hive-upgrade-acid-3.1.1.jar:/opt/CDS/hive/lib/hive-vector-code-gen-3.1.1.jar:/opt/CDS/hive/lib/hk2-api-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-locator-2.5.0-b32.jar:/opt/CDS/hive/lib/hk2-utils-2.5.0-b32.jar:/opt/CDS/hive/lib/hppc-0.7.2.jar:/opt/CDS/hive/lib/htrace-core-3.2.0-incubating.jar:/opt/CDS/hive/lib/httpclient-4.5.2.jar:/opt/CDS/hive/lib/httpcore-4.4.4.jar:/opt/CDS/hive/lib/ivy-2.4.0.jar:/opt/CDS/hive/lib/jackson-annotations-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-2.9.5.jar:/opt/CDS/hive/lib/jackson-core-asl-1.9.13.jar:/opt/CDS/hive/lib/jackson-databind-2.9.5.jar:/opt/CDS/hive/lib/jackson-dataformat-smile-2.9.5.jar:/opt/CDS/hive/lib/jackson-mapper-asl-1.9.13.jar:/opt/CDS/hive/lib/jamon-runtime-2.3.1.jar:/opt/CDS/hive/lib/janino-2.7.6.jar:/opt/CDS/hive/lib/javassist-3.20.0-GA.jar:/opt/CDS/hive/lib/javax.annotation-api-1.2.jar:/opt/CDS/hive/lib/javax.inject-2.5.0-b32.jar:/opt/CDS/hive/lib/javax.jdo-3.2.0-m3.jar:/opt/CDS/hive/lib/javax.servlet-api-3.1.0.jar:/opt/CDS/hive/lib/javax.servlet.jsp-2.3.2.jar:/opt/CDS/hive/lib/javax.servlet.jsp-api-2.3.1.jar:/opt/CDS/hive/lib/javax.ws.rs-api-2.0.1.jar:/opt/CDS/hive/lib/javolution-5.5.1.jar:/opt/CDS/hive/lib/jcodings-1.0.18.jar:/opt/CDS/hive/lib/jcommander-1.32.jar:/opt/CDS/hive/lib/jdo-api-3.0.1.jar:/opt/CDS/hive/lib/jersey-client-2.25.1.jar:/opt/CDS/hive/lib/jersey-common-2.25.1.jar:/opt/CDS/hive/lib/jersey-container-servlet-core-2.25.1.jar:/opt/CDS/hive/lib/jersey-guava-2.25.1.jar:/opt/CDS/hive/lib/jersey-media-jaxb-2.25.1.jar:/opt/CDS/hive/lib/jersey-server-2.25.1.jar:/opt/CDS/hive/lib/jettison-1.1.jar:/opt/CDS/hive/lib/jetty-annotations-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-http-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-io-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jaas-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-jndi-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-plus-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-rewrite-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-runner-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-schemas-3.1.jar:/opt/CDS/hive/lib/jetty-security-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-util-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-webapp-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jetty-xml-9.3.20.v20170531.jar:/opt/CDS/hive/lib/jline-2.12.jar:/opt/CDS/hive/lib/joda-time-2.9.9.jar:/opt/CDS/hive/lib/joni-2.1.11.jar:/opt/CDS/hive/lib/jpam-1.1.jar:/opt/CDS/hive/lib/json-1.8.jar:/opt/CDS/hive/lib/jsr305-3.0.0.jar:/opt/CDS/hive/lib/jta-1.1.jar:/opt/CDS/hive/lib/kryo-shaded-4.0.2.jar:/opt/CDS/hive/lib/libfb303-0.9.3.jar:/opt/CDS/hive/lib/libthrift-0.9.3.jar:/opt/CDS/hive/lib/memory-0.9.0.jar:/opt/CDS/hive/lib/metrics-core-3.1.0.jar:/opt/CDS/hive/lib/metrics-json-3.1.0.jar:/opt/CDS/hive/lib/metrics-jvm-3.1.0.jar:/opt/CDS/hive/lib/minlog-1.3.0.jar:/opt/CDS/hive/lib/mysql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/netty-3.10.5.Final.jar:/opt/CDS/hive/lib/netty-all-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-buffer-4.1.17.Final.jar:/opt/CDS/hive/lib/netty-common-4.1.17.Final.jar:/opt/CDS/hive/lib/opencsv-2.3.jar:/opt/CDS/hive/lib/orc-core-1.5.1.jar:/opt/CDS/hive/lib/orc-shims-1.5.1.jar:/opt/CDS/hive/lib/org.abego.treelayout.core-1.0.1.jar:/opt/CDS/hive/lib/osgi-resource-locator-1.0.1.jar:/opt/CDS/hive/lib/paranamer-2.3.jar:/opt/CDS/hive/lib/parquet-hadoop-bundle-1.10.0.jar:/opt/CDS/hive/lib/postgresql-9.4.1208.jre7.jar:/opt/CDS/hive/lib/postgresql-metadata-storage-0.12.0.jar:/opt/CDS/hive/lib/protobuf-java-2.5.0.jar:/opt/CDS/hive/lib/scala-library-2.11.12.jar:/opt/CDS/hive/lib/sketches-core-0.9.0.jar:/opt/CDS/hive/lib/snappy-java-1.1.4.jar:/opt/CDS/hive/lib/spark-core_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-network-common_2.11-2.4.0.jar:/opt/CDS/hive/lib/spark-unsafe_2.11-2.4.0.jar:/opt/CDS/hive/lib/sqlline-1.3.0.jar:/opt/CDS/hive/lib/stax-api-1.0.1.jar:/opt/CDS/hive/lib/super-csv-2.2.0.jar:/opt/CDS/hive/lib/taglibs-standard-impl-1.2.5.jar:/opt/CDS/hive/lib/taglibs-standard-spec-1.2.5.jar:/opt/CDS/hive/lib/tempus-fugit-1.1.jar:/opt/CDS/hive/lib/transaction-api-1.1.jar:/opt/CDS/hive/lib/validation-api-1.1.0.Final.jar:/opt/CDS/hive/lib/velocity-1.5.jar:/opt/CDS/hive/lib/websocket-api-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-client-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-common-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-server-9.3.20.v20170531.jar:/opt/CDS/hive/lib/websocket-servlet-9.3.20.v20170531.jar:/opt/CDS/hive/lib/zookeeper-3.4.6.jar:/opt/CDS/hadoop-2.9.2/share/hadoop/tools/lib/hadoop-distcp-2.9.2.jar:/opt/CDS/hbase//conf/:/opt/CDS/hbase//lib/shaded-clients/hbase-shaded-mapreduce-2.1.1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/commons-logging-1.2.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/log4j-1.2.17.jar:/opt/CDS/hbase//lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/opt/CDS/hive/lib/log4j-1.2-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-api-2.10.0.jar:/opt/CDS/hive/lib/log4j-core-2.10.0.jar:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar:/opt/CDS/hive/lib/log4j-web-2.10.0.jar:/opt/CDS/hadoop-2.9.2/contrib/capacity-scheduler/*.jar:/opt/CDS/hadoop-2.9.2//share/hadoop/common/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/common/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/hdfs/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/lib/*:/opt/CDS/hadoop-2.9.2/share/hadoop/yarn/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/lib/*:/opt/CDS/hadoop-2.9.2//share/hadoop/mapreduce/*" "-Xmx4096M" "-Dspark.hadoop.hbase.http.filter.initializers=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter" "-Dspark.hadoop.hbase.defaults.for.version=2.0.0-alpha4" "-Dspark.hadoop.hbase.wal.dir.perms=700" "-Dspark.hadoop.hbase.hstore.checksum.algorithm=CRC32C" "-Dspark.hadoop.hbase.regionserver.thrift.framed.max_frame_size_in_mb=2" "-Dspark.hadoop.hbase.snapshot.restore.take.failsafe.snapshot=true" "-Dspark.hadoop.hbase.regionserver.port=60020" "-Dspark.hadoop.hbase.rpc.rows.warning.threshold=5000" "-Dspark.hadoop.hbase.storescanner.parallel.seek.enable=false" "-Dspark.hadoop.hbase.client.max.perregion.tasks=1" "-Dspark.hadoop.hbase.mob.compaction.threads.max=1" "-Dspark.hadoop.hbase.mob.compaction.mergeable.threshold=1342177280" "-Dspark.hadoop.hbase.rpc.timeout=60000" "-Dspark.hadoop.hbase.replication.rpc.codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags" "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer" "-Dspark.hadoop.hbase.auth.token.max.lifetime=604800000" "-Dspark.hadoop.hbase.mob.compaction.chore.period=604800" "-Dspark.hadoop.hbase.bulkload.retries.number=10" "-Dspark.hadoop.hbase.hregion.memstore.flush.size=134217728" "-Dspark.master=spark://172.30.61.226:7077 <http://172.30.61.226:7077/>" "-Dspark.hadoop.hbase.master.port=60000" "-Dspark.hadoop.hbase.server.versionfile.writeattempts=3" "-Dspark.hadoop.hbase.regionserver.logroll.errors.tolerated=2" "-Dspark.hadoop.hbase.unsafe.stream.capability.enforce=false" "-Dspark.hadoop.hbase.master.info.port=60010" "-Dspark.app.name <http://dspark.app.name/>=Hive on Spark (sessionId = 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)" "-Dspark.hadoop.hbase.table.lock.enable=true" "-Dspark.hadoop.hbase.lease.recovery.dfs.timeout=64000" "-Dspark.hadoop.hbase.regionserver.thrift.compact=false" "-Dspark.hadoop.hbase.hregion.majorcompaction=604800000" "-Dspark.hadoop.hbase.zookeeper.peerport=2888" "-Dspark.hadoop.hbase.cluster.distributed=true" "-Dspark.hadoop.hbase.regionserver.storefile.refresh.period=0" "-Dspark.kryo.classesToRegister=org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,org.apache.hadoop.io.Writable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.io.HiveKey" "-Dspark.hadoop.hbase.regionserver.optionalcacheflushinterval=3600000" "-Dspark.hadoop.hbase.master.procedurewalcleaner.ttl=604800000" "-Dspark.hadoop.hbase.zookeeper.quorum=172.30.61.229,172.30.61.230,172.30.61.231" "-Dspark.sql.warehouse.dir=/hive/cdswarehouse" "-Dspark.hadoop.fs.defaultFS=hdfs://cdscluster/" "-Dspark.hadoop.hbase.hstore.blockingStoreFiles=10" "-Dspark.hadoop.hbase.regionserver.hlog.reader.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader" "-Dspark.hadoop.hbase.regionserver.checksum.verify=true" "-Dspark.hadoop.hbase.rest.readonly=false" "-Dspark.hadoop.hbase.regionserver.majorcompaction.pagecache.drop=true" "-Dspark.submit.deployMode=cluster" "-Dspark.hadoop.hbase.http.max.threads=16" "-Dspark.hadoop.hbase.data.umask.enable=false" "-Dspark.hadoop.hbase.hstore.flusher.count=2" "-Dspark.hadoop.hbase.server.compactchecker.interval.multiplier=1000" "-Dspark.hadoop.hbase.hstore.compaction.max.size=9223372036854775807" "-Dspark.hadoop.hbase.rest.filter.classes=org.apache.hadoop.hbase.rest.filter.GzipFilter" "-Dspark.hadoop.hbase.rest.csrf.enabled=false" "-Dspark.hadoop.hbase.coordinated.state.manager.class=org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager" "-Dspark.hadoop.hbase.mob.delfile.max.count=3" "-Dspark.hadoop.hbase.master.loadbalancer.class=org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer" "-Dspark.hadoop.hbase.snapshot.region.timeout=300000" "-Dspark.hadoop.hbase.hstore.compactionThreshold=3" "-Dspark.hadoop.hbase.data.umask=000" "-Dspark.client.authentication.secret=0ffffffea690fffffffb4d0ffffffc045240ffffffff410ffffffd56874181d04700ffffffae320ffffffc40ffffffe036c1a321c63e5e0ffffffde213a3d" "-Dspark.hadoop.hbase.hstore.bytes.per.checksum=16384" "-Dspark.hadoop.hbase.server.scanner.max.result.size=104857600" "-Dspark.hadoop.hbase.ipc.server.callqueue.scan.ratio=0" "-Dspark.hadoop.hbase.ipc.client.tcpnodelay=true" "-Dspark.hadoop.hbase.hstore.compaction.ratio.offpeak=5.0F" "-Dspark.hadoop.hbase.zookeeper.property.dataDir=/data/CDS/zookeeper" "-Dspark.jars=file:/opt/CDS/hive/lib/hive-exec-3.1.1.jar" "-Dspark.hadoop.hbase.client.pause=100" "-Dspark.hadoop.hbase.coprocessor.enabled=true" "-Dspark.hadoop.hbase.thrift.minWorkerThreads=16" "-Dspark.kryo.referenceTracking=false" "-Dspark.hadoop.hbase.master.mob.ttl.cleaner.period=86400" "-Dspark.hadoop.hbase.status.published=false" "-Dspark.hadoop.hbase.client.retries.number=35" "-Dspark.hadoop.hbase.hstore.blockingWaitTime=90000" "-Dspark.hadoop.hbase.mob.cache.evict.remain.ratio=0.5f" "-Dspark.hadoop.hbase.master.normalizer.class=org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer" "-Dspark.hadoop.hbase.client.scanner.caching=2147483647" "-Dspark.hadoop.hbase.normalizer.period=300000" "-Dspark.hadoop.hbase.hregion.max.filesize=10737418240" "-Dspark.hadoop.hbase.hregion.memstore.mslab.enabled=true" "-Dspark.hadoop.hbase.hstore.compaction.ratio=1.2F" "-Dspark.hadoop.hbase.security.visibility.mutations.checkauths=false" "-Dspark.hadoop.hbase.client.localityCheck.threadPoolSize=2" "-Dspark.hadoop.hbase.master.hfilecleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner" "-Dspark.hadoop.hbase.coprocessor.user.enabled=true" "-Dspark.hadoop.hbase.rs.cacheblocksonwrite=false" "-Dspark.executor.memory=6048m" "-Dspark.hadoop.hbase.regionserver.msginterval=3000" "-Dspark.hadoop.hbase.snapshot.master.timeout.millis=300000" "-Dspark.executor.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/" "-Dspark.hadoop.cloneConf=true" "-Dspark.hadoop.hbase.fs.tmp.dir=/user/cdsuser/hbase-staging" "-Dspark.hadoop.hbase.mob.cache.evict.period=3600" "-Dspark.hadoop.hbase.security.exec.permission.checks=false" "-Dspark.hadoop.hbase.hregion.percolumnfamilyflush.size.lower.bound.min=16777216" "-Dspark.hadoop.hbase.zookeeper.property.syncLimit=5" "-Dspark.history.fs.logDirectory=hdfs://cdscluster/spark-logs" "-Dspark.hadoop.hbase.client.operation.timeout=1200000" "-Dspark.hadoop.hbase.ipc.server.callqueue.handler.factor=0.1" "-Dspark.hadoop.hbase.rootdir=hdfs://cdscluster/hbase" "-Dspark.hadoop.hbase.ipc.server.callqueue.read.ratio=0" "-Dspark.eventLog.enabled=true" "-Dspark.hadoop.hbase.regionserver.handler.abort.on.error.percent=0.5" "-Dspark.hadoop.hbase.thrift.maxWorkerThreads=1000" "-Dspark.hadoop.hbase.regionserver.logroll.period=3600000" "-Dspark.hadoop.hbase.mob.file.cache.size=1000" "-Dspark.hadoop.hbase.master.logcleaner.plugins=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner" "-Dspark.hadoop.hbase.coprocessor.abortonerror=true" "-Dspark.hadoop.hbase.hregion.preclose.flush.size=5242880" "-Dspark.hadoop.hbase.dynamic.jars.dir=hdfs://cdscluster/hbase/lib" "-Dspark.hadoop.hbase.column.max.version=1" "-Dspark.hadoop.hbase.master.logcleaner.ttl=600000" "-Dspark.hadoop.hbase.regionserver.hostname.disable.master.reversedns=false" "-Dspark.hadoop.hbase.replication.source.maxthreads=10" "-Dspark.hadoop.hbase.client.scanner.timeout.period=60000" "-Dspark.hadoop.hbase.security.authentication=simple" "-Dspark.hadoop.hbase.snapshot.enabled=true" "-Dspark.hadoop.hbase.master.balancer.maxRitPercent=1.0" "-Dspark.hadoop.hbase.zookeeper.dns.interface=default" "-Dspark.driver.extraJavaOptions=-Dhive.spark.log.dir=/opt/CDS/spark/logs/" "-Dspark.hadoop.hbase.display.keys=true" "-Dspark.hadoop.hbase.regionserver.info.port.auto=false" "-Dspark.hadoop.hbase.client.write.buffer=2097152" "-Dspark.hadoop.hbase.regionserver.minorcompaction.pagecache.drop=true" "-Dspark.hadoop.hbase.server.keyvalue.maxsize=10485760" "-Dspark.history.fs.update.interval=10s" "-Dspark.hadoop.hbase.rest.threads.max=100" "-Dspark.hadoop.hbase.systemtables.compacting.memstore.type=NONE" "-Dspark.hadoop.hbase.client.keyvalue.maxsize=10485760" "-Dspark.hadoop.hbase.client.max.total.tasks=100" "-Dspark.hadoop.hbase.defaults.for.version.skip=false" "-Dspark.hadoop.hbase.zookeeper.property.initLimit=10" "-Dspark.hadoop.hbase.mob.compaction.batch.size=100" "-Dspark.hadoop.hbase.master.info.bindAddress=0.0.0.0" "-Dspark.hadoop.hbase.regionserver.handler.count=30" "-Dspark.hadoop.hbase.regionserver.region.split.policy=org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy" "-Dspark.hadoop.hbase.rpc.shortoperation.timeout=10000" "-Dspark.hadoop.hbase.zookeeper.property.clientPort=2181" "-Dspark.hadoop.hbase.master.infoserver.redirect=true" "-Dspark.hadoop.hbase.hstore.compaction.min.size=134217728" "-Dspark.hadoop.hbase.hstore.compaction.max=10" "-Dspark.hadoop.hbase.thrift.maxQueuedRequests=1000" "-Dspark.driver.supervise=false" "-Dspark.sql.catalogImplementation=hive" "-Dspark.hadoop.hbase.local.dir=/tmp/hbase-cdsuser/local/" "-Dspark.hadoop.hbase.ipc.client.fallback-to-simple-auth-allowed=false" "-Dspark.hadoop.hbase.hstore.compaction.kv.max=10" "-Dspark.hadoop.hbase.regionserver.info.bindAddress=0.0.0.0" "-Dspark.hadoop.hbase.hregion.memstore.block.multiplier=4" "-Dspark.hadoop.hbase.regionserver.regionSplitLimit=1000" "-Dspark.hadoop.hbase.bucketcache.combinedcache.enabled=true" "-Dspark.eventLog.dir=/tmp" "-Dspark.hadoop.hbase.regionserver.dns.interface=default" "-Dspark.hadoop.hbase.storescanner.parallel.seek.threads=10" "-Dspark.hadoop.hbase.rest.port=8080" "-Dspark.hadoop.hbase.http.staticuser.user=dr.stack" "-Dspark.hadoop.hbase.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*" "-Dspark.client.authentication.client_id=9a1af6d4-59d9-49fc-b8ab-40469737cc9c" "-Dspark.hadoop.hbase.ipc.server.fallback-to-simple-auth-allowed=false" "-Dspark.hadoop.hbase.regionserver.hlog.writer.impl=org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter" "-Dspark.driver.memory=4g" "-Dspark.hadoop.hbase.client.perserver.requests.threshold=2147483647" "-Dspark.hadoop.hbase.regionserver.dns.nameserver=default" "-Dspark.hadoop.hbase.client.scanner.max.result.size=2097152" "-Dspark.hadoop.hbase.auth.key.update.interval=86400000" "-Dspark.hadoop.hbase.balancer.period=300000" "-Dspark.hadoop.hbase.snapshot.restore.failsafe.name <http://dspark.hadoop.hbase.snapshot.restore.failsafe.name/>=hbase-failsafe-{snapshot.name <http://snapshot.name/>}-{restore.timestamp}" "-Dspark.hadoop.hbase.status.publisher.class=org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher" "-Dspark.hadoop.hbase.rest.support.proxyuser=false" "-Dspark.hadoop.hbase.regionserver.thrift.framed=false" "-Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider" "-Dspark.hadoop.hbase.status.multicast.address.port=16100" "-Dspark.hadoop.hbase.master.hostname=nl-itc-l-cdma00002.labs.liberty.biz <http://nl-itc-l-cdma00002.labs.liberty.biz/>" "-Dspark.rpc.askTimeout=10s" "-Dspark.hadoop.hbase.client.max.perserver.tasks=2" "-Dspark.hadoop.hbase.serial.replication.waitingMs=10000" "-Dspark.hadoop.hbase.mob.compactor.class=org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactor" "-Dspark.history.ui.port=18080" "-Dspark.hadoop.hbase.zookeeper.property.maxClientCnxns=300" "-Dspark.hadoop.hbase.regions.slop=0.001" "-Dspark.hadoop.hbase.tmp.dir=/tmp/hbase-cdsuser" "-Dspark.hadoop.hbase.offpeak.end.hour=-1" "-Dspark.hadoop.zookeeper.znode.acl.parent=acl" "-Dspark.hadoop.hbase.server.thread.wakefrequency=10000" "-Dspark.hadoop.hbase.hregion.majorcompaction.jitter=0.50" "-Dspark.hadoop.hbase.region.replica.replication.enabled=false" "-Dspark.hadoop.hbase.zookeeper.leaderport=3888" "-Dspark.hadoop.hbase.regionserver.thread.compaction.throttle=2684354560" "-Dspark.hadoop.hbase.offpeak.start.hour=-1" "-Dspark.hadoop.hbase.rest.threads.min=2" "-Dspark.hadoop.hbase.regionserver.info.port=60030" "-Dspark.hadoop.hbase.lease.recovery.timeout=900000" "-Dspark.executor.cores=6" "-Dspark.hadoop.hbase.rootdir.perms=700" "-Dspark.hadoop.hbase.status.listener.class=org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener" "-Dspark.hadoop.hbase.hstore.time.to.purge.deletes=0" "-Dspark.hadoop.hbase.zookeeper.dns.nameserver=default" "-Dspark.hadoop.hbase.table.max.rowsize=1073741824" "-Dspark.hadoop.hbase.hstore.compaction.min=3" "-Dspark.hadoop.hbase.dfs.client.read.shortcircuit.buffer.size=131072" "-Dspark.hadoop.hbase.cells.scanned.per.heartbeat.check=10000" "-Dspark.hadoop.zookeeper.znode.parent=/hbase" "-Dspark.hadoop.hbase.status.multicast.address.ip=226.1.1.3" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@172.30.61.226:41266 <http://Worker@172.30.61.226:41266/>" "/opt/CDS/spark/work/driver-20220527123634-0000/hive-exec-3.1.1.jar" "org.apache.hive.spark.client.RemoteDriver" "--remote-host" "nl-itc-l-cdma00001.labs.liberty.biz <http://nl-itc-l-cdma00001.labs.liberty.biz/>" "--remote-port" "36667" "--conf" "hive.spark.client.connect.timeout=30000" "--conf" "hive.spark.client.server.connect.timeout=300000" "--conf" "hive.spark.client.channel.log.level=null" "--conf" "hive.spark.client.rpc.max.size=52428800" "--conf" "hive.spark.client.rpc.threads=8" "--conf" "hive.spark.client.secret.bits=256" "--conf" "hive.spark.client.rpc.server.address=172.30.61.226"
> ========================================
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/opt/CDS/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/CDS/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/CDS/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
> at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> Caused by: java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
> at org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:48)
> at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:138)
> at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:536)
> ... 6 more
>
> and eventually timesout
>
> following is the content of spark-defaults.conf
> ######################################################################################################################
> spark.master spark://172.30.61.226:7077 <http://172.30.61.226:7077/>
> spark.eventLog.enabled true
> spark.eventLog.dir hdfs://cdscluster/spark/
> spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
> spark.history.fs.logDirectory hdfs://cdscluster/spark-logs
> spark.history.fs.update.interval 10s
> spark.history.ui.port 18080
> spark.sql.catalogImplementation=hive
> spark.sql.warehouse.dir /hive/cdswarehouse
> spark.serializer org.apache.spark.serializer.KryoSerializer
> ######################################################################################################################
> hive-site.xml
> <?xml version="1.0" encoding="UTF-8" standalone="no"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <configuration>
>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>/hive/cdswarehouse</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:derby://172.30.61.100:1527/cds_hive_metastoredb;create=true <http://172.30.61.100:1527/cds_hive_metastoredb;create=true></value>
> </property>
> <property>
> <name>datanucleus.schema.autoCreateTables</name>
> <value>true</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>org.apache.derby.jdbc.ClientDriver</value>
> <description>Driver class name for a JDBC metastore</description>
> </property>
> <property>
> <name>hive.server2.thrift.port</name>
> <value>10000</value>
> </property>
>
> <property>
> <name>hive.server2.enable.doAs</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.execution.engine</name>
> <value>spark</value>
> </property>
> <property>
> <name>spark.master</name>
> <value>spark://172.30.61.226:7077 <http://172.30.61.226:7077/></value>
> </property>
> <property>
> <name>hive.metastore.port</name>
> <value>9083</value>
> </property>
>
> <property>
> <name>mapreduce.input.fileinputformat.input.dir.recursive</name>
> <value>true</value>
> </property>
> <property>
> <name>spark.eventLog.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.spark.client.rpc.server.address</name>
> <value>172.30.61.226</value>
> </property>
>
> <property>
> <name>spark.eventLog.dir</name>
> <value>/tmp</value>
> </property>
> <property>
> <name>spark.executor.memory</name>
> <value>6048m</value>
> </property>
> <property>
> <name>spark.executor.cores</name>
> <value>6</value>
> </property>
>
> <property>
> <name>hive.spark.client.connect.timeout</name>
> <value>30000ms</value>
> </property>
> <property>
> <name>hive.spark.client.server.connect.timeout</name>
> <value>300000ms</value>
> </property>
> </configuration>
> ######################################################################################################################
> Have followed the instructions in [https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started <https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started>] and have linked the jars
>
> lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35 spark-core_2.11-2.4.0.jar -> /opt/CDS/spark/jars/spark-core_2.11-2.4.0.jar
> lrwxrwxrwx 1 cdsuser cdsuser 45 May 27 12:35 scala-library-2.11.12.jar -> /opt/CDS/spark/jars/scala-library-2.11.12.jar
> lrwxrwxrwx 1 cdsuser cdsuser 55 May 27 12:35 spark-network-common_2.11-2.4.0.jar -> /opt/CDS/spark/jars/spark-network-common_2.11-2.4.0.jar
> lrwxrwxrwx 1 cdsuser cdsuser 47 May 27 12:35 spark-unsafe_2.11-2.4.0.jar -> /opt/CDS/spark/jars/spark-unsafe_2.11-2.4.0.jar
> ######################################################################################################################
>
> Hive console
> ##################################
> hive> select count(*) from cds.spectrum_data;
> 2022-05-27 12:36:31,955 INFO [e32ede4e-2110-4aea-9068-40e4a4986f78 main] reducesink.VectorReduceSinkEmptyKeyOperator: VectorReduceSinkEmptyKeyOperator constructor vectorReduceSinkInfo org.apache.hadoop.hive.ql.plan.VectorReduceSinkInfo@62dfe152
> Query ID = cdsuser_20220527123629_e7e2aa6a-0223-4943-b387-a24f09e26aa8
> Total jobs = 1
> Launching Job 1 out of 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=<number>
> Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c)'
> FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 9a1af6d4-59d9-49fc-b8ab-40469737cc9c
> ############################################################################################################
> Would really appreciate it if you can guide me on this.
>
>
> --
> Live every day as if it were your last, because one of these days, it will be.
>
> Regards,
>
> Prasanth M Sasidharan