You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mich Talebzadeh <mi...@peridale.co.uk> on 2015/12/03 11:28:55 UTC
Building spark 1.3 from source code to work with Hive 1.2.1
Hi,
I have seen mails that state that the user has managed to build spark 1.3 to
work with Hive. I tried Spark 1.5.2 but no luck
I downloaded spark source 1.3 source code spark-1.3.0.tar and built it as
follows
./make-distribution.sh --name "hadoop2-without-hive" --tgz
"-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"
This successfully completed and created the tarred zip file. I then created
spark 1.3 tree from this zipped file. $SPARK_HOME is /usr/lib/spark
Other steps that I performed:
1. In $HIVE_HOME/lib , I copied spark-assembly-1.3.0-hadoop2.4.0.jar to
this directory
2. In $SPARK_HOME/conf I created a syblink to
/usr/lib/hive/conf/hive-site.xml
Then I tried to start spark master node
/usr/lib/spark/sbin/start-master.sh
I get the following error:
cat
/usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Mast
er-1-rhes564.out
Spark Command: /usr/java/latest/bin/java -cp
:/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2
.4.0.jar:/home/hduser/hadoop-2.6.0/etc/hadoop -XX:MaxPermSize=128m
-Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m
org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port
8080
========================================
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
at java.lang.Class.getMethod0(Class.java:2764)
at java.lang.Class.getMethod(Class.java:1653)
at
sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at
sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I also notice that in /usr/lib/spark/lib, I only have the following jar
files
-rw-r--r-- 1 hduser hadoop 98795479 Dec 3 09:03
spark-examples-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 98187168 Dec 3 09:03
spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 4136760 Dec 3 09:03
spark-1.3.0-yarn-shuffle.jar
Wheras in pre-build downloaded one --> /usr/lib/spark-1.3.0-bin-hadoop2.4,
there are additional JAR files
-rw-rw-r-- 1 hduser hadoop 1890075 Mar 6 2015
datanucleus-core-3.2.10.jar
-rw-rw-r-- 1 hduser hadoop 112446389 Mar 6 2015
spark-examples-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 159319006 Mar 6 2015
spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 4136744 Mar 6 2015
spark-1.3.0-yarn-shuffle.jar
-rw-rw-r-- 1 hduser hadoop 1809447 Mar 6 2015
datanucleus-rdbms-3.2.9.jar
-rw-rw-r-- 1 hduser hadoop 339666 Mar 6 2015
datanucleus-api-jdo-3.2.6.jar
Any ideas what is is missing? I am sure someone has sorted this one out
before.
Thanks,
Mich
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.
Re: Building spark 1.3 from source code to work with Hive 1.2.1
Posted by Xuefu Zhang <xz...@cloudera.com>.
As mentioned a few times in the past, you need to build your
spark-assembly.jar without Hive and copied it to Hive's /lib directory. You
shouldn't use the one you downloaded from Spark installation for Hive to
use. The downloaded ones has Hive classes possibly of a different version.
--Xuefu
On Thu, Dec 3, 2015 at 8:43 AM, Mich Talebzadeh <mi...@peridale.co.uk> wrote:
> Thanks.
>
>
>
> I have done this before but little joy I am afraid and again errors!! Very
> disappointing
>
>
>
> 1. Downloaded *pre-build spark-13.0* and installed it.
>
> 2. Went to $HIVE_HOME/lib directory and created a symlink to
> $SPARK_HOME/lib/spark-assembly-1.3.0-hadoop2.4.0.jar
>
> 3. Went to $SPARK_HOME/conf directory and created a symlink to
> $HIVE_HOME/conf/hive-site.xml
>
> 4. Started spark-master
>
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
>
> Spark Command: /usr/java/latest/bin/java -cp
> :/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/usr/lib/spark/lib/datanucleus-core-3.2.10.jar:/usr/lib/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/lib/spark/lib/datanucleus-rdbms-3.2.9.jar:/home/hduser/hadoop-2.6.0/etc/hadoop
> -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m
> org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port
> 8080
>
> ========================================
>
>
>
> 15/12/03 15:35:53 INFO master.Master: Registered signal handlers for
> [TERM, HUP, INT]
>
> 15/12/03 15:35:54 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 15/12/03 15:35:54 INFO spark.SecurityManager: Changing view acls to: hduser
>
> 15/12/03 15:35:54 INFO spark.SecurityManager: Changing modify acls to:
> hduser
>
> 15/12/03 15:35:54 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hduser); users with modify permissions: Set(hduser)
>
> 15/12/03 15:35:54 INFO slf4j.Slf4jLogger: Slf4jLogger started
>
> 15/12/03 15:35:54 INFO Remoting: Starting remoting
>
> 15/12/03 15:35:54 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkMaster@rhes564:7077]
>
> 15/12/03 15:35:54 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkMaster@rhes564:7077]
>
> 15/12/03 15:35:54 INFO util.Utils: Successfully started service
> 'sparkMaster' on port 7077.
>
> 15/12/03 15:35:55 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/12/03 15:35:55 INFO server.AbstractConnector: Started
> SelectChannelConnector@rhes564:6066
>
> 15/12/03 15:35:55 INFO util.Utils: Successfully started service on port
> 6066.
>
> 15/12/03 15:35:55 INFO rest.StandaloneRestServer: Started REST server for
> submitting applications on port 6066
>
> 15/12/03 15:35:55 INFO master.Master: Starting Spark master at
> spark://rhes564:7077
>
> 15/12/03 15:35:55 INFO master.Master: Running Spark version 1.3.0
>
> 15/12/03 15:35:55 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/12/03 15:35:55 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:8080
>
> 15/12/03 15:35:55 INFO util.Utils: Successfully started service 'MasterUI'
> on port 8080.
>
> 15/12/03 15:35:55 INFO ui.MasterWebUI: Started MasterWebUI at
> http://rhes564:8080
>
> 15/12/03 15:35:55 INFO master.Master: I have been elected leader! New
> state: ALIVE
>
> 15/12/03 15:36:58 INFO master.Master: Registering worker rhes564:56498
> with 12 cores, 22.5 GB RAM
>
> 5. Started a worker process as start-slave.sh 1 spark://rhes564:7077
>
> 6. Went back to Hive in debug mode and did
>
> set spark.home=/usr/lib/spark;
>
> set hive.execution.engine=spark;
>
> set spark.master=spark://rhes564:7077;
>
> set spark.eventLog.enabled=true;
>
> set spark.eventLog.dir=/usr/lib/spark/logs;
>
> set spark.executor.memory=512m;
>
> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
>
> set spark.home=/usr/lib/spark;
>
> 15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution
> is on: /usr/lib/spark
>
> hive> set hive.execution.engine=spark;
>
> 15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution
> is on: spark
>
> hive> set spark.master=spark://rhes564:7077;
>
> 15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution
> is on: spark://rhes564:7077
>
> hive> set spark.eventLog.enabled=true;
>
> 15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution
> is on: true
>
> hive> set spark.eventLog.dir=/usr/lib/spark/logs;
>
> 15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution
> is on: /usr/lib/spark/logs
>
> hive> set spark.executor.memory=512m;
>
> 15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution
> is on: 512m
>
> hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
>
> 15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution
> is on: org.apache.spark.serializer.KryoSerializer
>
> *7. **select count(1) from t;*
>
> 15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=Driver.run
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG
> method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=compile
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:53 [main]: DEBUG parse.VariableSubstitution: Substitution
> is on: select count(1) from t
>
> 15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=parse
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:53 [main]: INFO parse.ParseDriver: Parsing command: select
> count(1) from t
>
> 15/12/03 16:18:53 [main]: INFO parse.ParseDriver: Parse Completed
>
> 15/12/03 16:18:53 [main]: INFO log.PerfLogger: </PERFLOG method=parse
> start=1449159533904 end=1449159533916 duration=12
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:53 [main]: DEBUG ql.Driver: Encoding valid txns info
> 9223372036854775807:
>
> 15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG
> method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Starting Semantic
> Analysis
>
> 15/12/03 16:18:53 [main]: DEBUG exec.FunctionRegistry: Looking up
> GenericUDAF: count
>
> 15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Completed phase 1 of
> Semantic Analysis
>
> 15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Get metadata for
> source tables
>
> 15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Get metadata for
> subqueries
>
> 15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Get metadata for
> destination tables
>
> 15/12/03 16:18:54 [main]: DEBUG hdfs.DFSClient:
> /tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1:
> masked=rwx------
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.Client: Connecting to rhes564/
> 50.140.197.217:9000
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser:
> starting, having connections 1
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #9
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #9
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: mkdirs took
> 6ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #10
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #10
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 1ms
>
> 15/12/03 16:18:54 [main]: INFO ql.Context: New scratch dir is
> hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1
>
> 15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Completed getting
> MetaData in Semantic Analysis
>
> 15/12/03 16:18:54 [main]: INFO parse.BaseSemanticAnalyzer: Not invoking
> CBO because the statement has too few joins
>
> 15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30)
> owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id,
> i64 data_object_id, varchar(19) object_type, timestamp created, timestamp
> last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1)
> temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace,
> varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2,
> varchar(32) attribute, i32 op_type, timestamp op_time}
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Table Plan
> for t TS[0]
>
> 15/12/03 16:18:54 [main]: DEBUG exec.FunctionRegistry: Looking up
> GenericUDAF: count
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: RR before GB
> t{(owner,owner: varchar(30))(object_name,object_name:
> varchar(30))(subobject_name,subobject_name:
> varchar(30))(object_id,object_id: bigint)(data_object_id,data_object_id:
> bigint)(object_type,object_type: varchar(19))(created,created:
> timestamp)(last_ddl_time,last_ddl_time: timestamp)(timestamp,timestamp:
> varchar(19))(status,status: varchar(7))(temporary2,temporary2:
> varchar(1))(generated,generated: varchar(1))(secondary,secondary:
> varchar(1))(namespace,namespace: bigint)(edition_name,edition_name:
> varchar(30))(padding1,padding1: varchar(4000))(padding2,padding2:
> varchar(3500))(attribute,attribute: varchar(32))(op_type,op_type:
> int)(op_time,op_time:
> timestamp)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE:
> bigint)(input__file__name,INPUT__FILE__NAME: string)(row__id,ROW__ID:
> struct<transactionid:bigint,bucketid:int,rowid:bigint>)} after GB
> {((tok_function count 1),_col0: bigint)}
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: tree: (TOK_SELECT
> (TOK_SELEXPR (TOK_FUNCTION count 1)))
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: genSelectPlan: input
> = {((tok_function count 1),_col0: bigint)} starRr = t{(owner,owner:
> varchar(30))(object_name,object_name:
> varchar(30))(subobject_name,subobject_name:
> varchar(30))(object_id,object_id: bigint)(data_object_id,data_object_id:
> bigint)(object_type,object_type: varchar(19))(created,created:
> timestamp)(last_ddl_time,last_ddl_time: timestamp)(timestamp,timestamp:
> varchar(19))(status,status: varchar(7))(temporary2,temporary2:
> varchar(1))(generated,generated: varchar(1))(secondary,secondary:
> varchar(1))(namespace,namespace: bigint)(edition_name,edition_name:
> varchar(30))(padding1,padding1: varchar(4000))(padding2,padding2:
> varchar(3500))(attribute,attribute: varchar(32))(op_type,op_type:
> int)(op_time,op_time:
> timestamp)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE:
> bigint)(input__file__name,INPUT__FILE__NAME: string)(row__id,ROW__ID:
> struct<transactionid:bigint,bucketid:int,rowid:bigint>)}
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Select Plan
> row schema: null{(_c0,_col0: bigint)}
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Select Plan
> for clause: insclause-0
>
> 15/12/03 16:18:54 [main]: DEBUG ql.Context: Created staging dir =
> hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1
> for path =
> hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000
>
> 15/12/03 16:18:54 [main]: INFO common.FileUtils: Creating directory if it
> doesn't exist:
> hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #11
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #11
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 1ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #12
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #12
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 0ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #13
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #13
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 1ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #14
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #14
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 0ms
>
> 15/12/03 16:18:54 [main]: DEBUG hdfs.DFSClient:
> /tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1:
> masked=rwxr-xr-x
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #15
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #15
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: mkdirs took
> 1ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #16
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #16
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 1ms
>
> 15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure:
> {-chgrp,-R,supergroup,hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000}
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #17
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #17
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 0ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #18
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #18
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing
> took 1ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #19
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #19
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing
> took 1ms
>
> 15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: Return value is :0
>
> 15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure:
> {-chmod,-R,700,hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000}
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #20
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #20
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 0ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #21
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #21
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: setPermission
> took 1ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #22
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #22
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing
> took 1ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #23
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #23
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: setPermission
> took 5ms
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #24
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #24
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing
> took 0ms
>
> 15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: Return value is :0
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #25
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #25
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 1ms
>
> 15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure:
> FileStatus{path=hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000;
> isDirectory=true; modification_time=1449159534181; access_time=0;
> owner=hduser; group=supergroup; permission=rwx------; isSymlink=false}
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #26
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #26
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo
> took 0ms
>
> 15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with:
> columnNames=[_col0] columnTypes=[bigint] separator=[[B@51669ea5]
> nullstring=\N lastColumnTakesRest=false timestampFormats=null
>
> 15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with:
> columnNames=[_col0] columnTypes=[bigint] separator=[[B@15a2b981]
> nullstring=\N lastColumnTakesRest=false timestampFormats=null
>
> 15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Set stats collection
> dir :
> hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1/-ext-10002
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created FileSink
> Plan for clause: insclause-0dest_path:
> hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000
> row schema: null{(_c0,_col0: bigint)}
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Body Plan
> for Query Block null
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Plan for
> Query Block null
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Before logical
> optimization
>
> TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-SEL[5]-FS[6]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:TS[0] with rs:(owner:
> varchar(30)|{t}owner,object_name:
> varchar(30)|{t}object_name,subobject_name:
> varchar(30)|{t}subobject_name,object_id:
> bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type:
> varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time:
> timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status:
> varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated:
> varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace:
> bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1:
> varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute:
> varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time:
> timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE:
> bigint|{t}block__offset__inside__file,INPUT__FILE__NAME:
> string|{t}input__file__name,ROW__ID:
> struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator TS[0]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:SEL[1] with rs:(owner:
> varchar(30)|{t}owner,object_name:
> varchar(30)|{t}object_name,subobject_name:
> varchar(30)|{t}subobject_name,object_id:
> bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type:
> varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time:
> timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status:
> varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated:
> varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace:
> bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1:
> varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute:
> varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time:
> timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE:
> bigint|{t}block__offset__inside__file,INPUT__FILE__NAME:
> string|{t}input__file__name,ROW__ID:
> struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator SEL[1]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory:
> New column list:(Column[owner] Column[object_name] Column[subobject_name]
> Column[object_id] Column[data_object_id] Column[object_type]
> Column[created] Column[last_ddl_time] Column[timestamp] Column[status]
> Column[temporary2] Column[generated] Column[secondary] Column[namespace]
> Column[edition_name] Column[padding1] Column[padding2] Column[attribute]
> Column[op_type] Column[op_time] Column[BLOCK__OFFSET__INSIDE__FILE]
> Column[INPUT__FILE__NAME] Column[ROW__ID])
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:GBY[2] with rs:(_col0: bigint|{}(tok_function count
> 1))
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator GBY[2]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:RS[3] with rs:(VALUE._col0: bigint|{}(tok_function
> count 1))
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator RS[3]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:GBY[4] with rs:(_col0: bigint|{}(tok_function count
> 1))
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator GBY[4]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:SEL[5] with rs:(_col0: bigint|{null}_c0)
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator SEL[5]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory:
> New column list:(Column[_col0])
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:FS[6] with rs:(_col0: bigint|{})
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator FS[6]
>
> 15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for FS(6)
>
> 15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for SEL(5)
>
> 15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for GBY(4)
>
> 15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for RS(3)
>
> 15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for GBY(2)
>
> 15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for SEL(1)
>
> 15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for TS(0)
>
> 15/12/03 16:18:54 [main]: DEBUG ppd.PredicatePushDown: After PPD:
>
> TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-SEL[5]-FS[6]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:TS[0] with rs:(owner:
> varchar(30)|{t}owner,object_name:
> varchar(30)|{t}object_name,subobject_name:
> varchar(30)|{t}subobject_name,object_id:
> bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type:
> varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time:
> timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status:
> varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated:
> varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace:
> bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1:
> varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute:
> varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time:
> timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE:
> bigint|{t}block__offset__inside__file,INPUT__FILE__NAME:
> string|{t}input__file__name,ROW__ID:
> struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator TS[0]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:SEL[1] with rs:(owner:
> varchar(30)|{t}owner,object_name:
> varchar(30)|{t}object_name,subobject_name:
> varchar(30)|{t}subobject_name,object_id:
> bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type:
> varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time:
> timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status:
> varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated:
> varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace:
> bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1:
> varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute:
> varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time:
> timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE:
> bigint|{t}block__offset__inside__file,INPUT__FILE__NAME:
> string|{t}input__file__name,ROW__ID:
> struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator SEL[1]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory:
> New column list:(Column[owner] Column[object_name] Column[subobject_name]
> Column[object_id] Column[data_object_id] Column[object_type]
> Column[created] Column[last_ddl_time] Column[timestamp] Column[status]
> Column[temporary2] Column[generated] Column[secondary] Column[namespace]
> Column[edition_name] Column[padding1] Column[padding2] Column[attribute]
> Column[op_type] Column[op_time] Column[BLOCK__OFFSET__INSIDE__FILE]
> Column[INPUT__FILE__NAME] Column[ROW__ID])
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:GBY[2] with rs:(_col0: bigint|{}(tok_function count
> 1))
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator GBY[2]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:RS[3] with rs:(VALUE._col0: bigint|{}(tok_function
> count 1))
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator RS[3]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:GBY[4] with rs:(_col0: bigint|{}(tok_function count
> 1))
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator GBY[4]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:SEL[5] with rs:(_col0: bigint|{null}_c0)
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator SEL[5]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory:
> New column list:(Column[_col0])
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Getting constants of op:FS[6] with rs:(_col0: bigint|{})
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx:
> Offerring constants [] to operator FS[6]
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.ColumnPrunerProcFactory: Reduce
> Sink Operator 3 key:[]
>
> 15/12/03 16:18:54 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 3
> oldColExprMap: {VALUE._col0=Column[_col0]}
>
> 15/12/03 16:18:54 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 3
> newColExprMap: {VALUE._col0=Column[_col0]}
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.IdentityProjectRemover: Identity
> project remover optimization removed : SEL[5]
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG
> method=partition-retrieving
> from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG
> method=partition-retrieving start=1449159534437 end=1449159534440
> duration=3 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
>
> 15/12/03 16:18:54 [main]: DEBUG exec.TableScanOperator: Setting stats (Num
> rows: 1767886 Data size: 5984968162 Basic stats: COMPLETE Column stats:
> COMPLETE) on TS[0]
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0]
> STATS-TS[0] (t): numRows: 1767886 dataSize: 5984968162 basicStatsState:
> COMPLETE colStatsState: COMPLETE colStats: {}
>
> 15/12/03 16:18:54 [main]: DEBUG exec.SelectOperator: Setting stats (Num
> rows: 1767886 Data size: 5984968162 Basic stats: COMPLETE Column stats:
> COMPLETE) on SEL[1]
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0]
> STATS-SEL[1]: numRows: 1767886 dataSize: 5984968162 basicStatsState:
> COMPLETE colStatsState: COMPLETE colStats: {}
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory:
> STATS-GBY[2]: inputSize: 5984968162 maxSplitSize: 256000000 parallelism: 24
> containsGroupingSet: false sizeOfGroupingSet: 1
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory:
> STATS-GBY[2] hashAgg: true
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [Case 3]
> STATS-GBY[2]: cardinality: 24
>
> 15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting stats (Num
> rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE) on GBY[2]
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0]
> STATS-GBY[2]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE
> colStatsState: COMPLETE colStats: {_col0= colName: _col0 colType: bigint
> countDistincts: 24 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0
> isPrimaryKey: false}
>
> 15/12/03 16:18:54 [main]: DEBUG exec.ReduceSinkOperator: Setting stats
> (Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE) on
> RS[3]
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0]
> STATS-RS[3]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE
> colStatsState: COMPLETE colStats: {VALUE._col0= colName: VALUE._col0
> colType: bigint countDistincts: 24 numNulls: 0 avgColLen: 8.0 numTrues: 0
> numFalses: 0 isPrimaryKey: false}
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory:
> STATS-GBY[4]: inputSize: 1 maxSplitSize: 256000000 parallelism: 1
> containsGroupingSet: false sizeOfGroupingSet: 1
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory:
> STATS-GBY[4] hashAgg: false
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [Case 9]
> STATS-GBY[4]: cardinality: 1
>
> 15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting stats (Num
> rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE) on GBY[4]
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0]
> STATS-GBY[4]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE
> colStatsState: COMPLETE colStats: {_col0= colName: _col0 colType: bigint
> countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0
> isPrimaryKey: false}
>
> 15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0]
> STATS-FS[6]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE
> colStatsState: COMPLETE colStats: {_col0= colName: _col0 colType: bigint
> countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0
> isPrimaryKey: false}
>
> 15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #27
>
> 15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #27
>
> 15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing
> took 1ms
>
> 15/12/03 16:18:54 [main]: DEBUG exec.TableScanOperator: Setting traits
> (org.apache.hadoop.hive.ql.plan.OpTraits@34dccb3d) on TS[0]
>
> 15/12/03 16:18:54 [main]: DEBUG exec.SelectOperator: Setting traits
> (org.apache.hadoop.hive.ql.plan.OpTraits@ba32ce8) on SEL[1]
>
> 15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting traits
> (org.apache.hadoop.hive.ql.plan.OpTraits@5d93950) on GBY[2]
>
> 15/12/03 16:18:54 [main]: DEBUG exec.ReduceSinkOperator: Setting traits
> (org.apache.hadoop.hive.ql.plan.OpTraits@7841b62f) on RS[3]
>
> 15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting traits
> (org.apache.hadoop.hive.ql.plan.OpTraits@22200557) on GBY[4]
>
> 15/12/03 16:18:54 [main]: DEBUG exec.FileSinkOperator: Setting traits
> (org.apache.hadoop.hive.ql.plan.OpTraits@22200557) on FS[6]
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: After logical
> optimization
>
> TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-FS[6]
>
> 15/12/03 16:18:54 [main]: DEBUG security.Groups: Returning cached groups
> for 'hduser'
>
> 15/12/03 16:18:54 [main]: DEBUG security.Groups: Returning cached groups
> for 'hduser'
>
> 15/12/03 16:18:54 [main]: DEBUG session.SessionState: Session is using
> authorization class class
> org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG
> method=SparkOptimizeOperatorTree
> from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
>
> 15/12/03 16:18:54 [main]: INFO spark.SetSparkReducerParallelism: Number of
> reducers determined to be: 1
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG
> method=SparkOptimizeOperatorTree start=1449159534470 end=1449159534477
> duration=7 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG
> method=SparkGenerateTaskTree
> from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Root operator: TS[0]
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Leaf operator: RS[3]
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkUtils: Adding map work (Map
> 1) for TS[0]
>
> 15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30)
> owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id,
> i64 data_object_id, varchar(19) object_type, timestamp created, timestamp
> last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1)
> temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace,
> varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2,
> varchar(32) attribute, i32 op_type, timestamp op_time}
>
> 15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30)
> owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id,
> i64 data_object_id, varchar(19) object_type, timestamp created, timestamp
> last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1)
> temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace,
> varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2,
> varchar(32) attribute, i32 op_type, timestamp op_time}
>
> 15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30)
> owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id,
> i64 data_object_id, varchar(19) object_type, timestamp created, timestamp
> last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1)
> temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace,
> varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2,
> varchar(32) attribute, i32 op_type, timestamp op_time}
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.GenMapRedUtils: Adding
> hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t of tablet
>
> 15/12/03 16:18:54 [main]: DEBUG optimizer.GenMapRedUtils: Information
> added for path hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: First pass. Leaf
> operator: RS[3]
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Root operator: GBY[4]
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Leaf operator: FS[6]
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkUtils: Adding reduce work
> (Reducer 2) for GBY[4]
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkUtils: Setting up reduce
> sink: RS[3] with following reduce work: Reducer 2
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Removing RS[3] as
> parent from GBY[4]
>
> 15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: First pass. Leaf
> operator: FS[6]
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG
> method=SparkGenerateTaskTree start=1449159534477 end=1449159534495
> duration=18 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG
> method=SparkOptimizeTaskTree
> from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
>
> 15/12/03 16:18:54 [main]: DEBUG parse.TaskCompiler: Skipping runtime skew
> join optimization
>
> 15/12/03 16:18:54 [main]: DEBUG physical.NullScanTaskDispatcher: Looking
> at: Map 1
>
> 15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Looking
> for table scans where optimization is applicable
>
> 15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Found 0
> null table scans
>
> 15/12/03 16:18:54 [main]: DEBUG physical.NullScanTaskDispatcher: Looking
> at: Map 1
>
> 15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Looking
> for table scans where optimization is applicable
>
> 15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Found 0
> null table scans
>
> 15/12/03 16:18:54 [main]: DEBUG physical.NullScanTaskDispatcher: Looking
> at: Map 1
>
> 15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Looking
> for table scans where optimization is applicable
>
> 15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Found 0
> null table scans
>
> 15/12/03 16:18:54 [main]: DEBUG parse.TaskCompiler: Skipping vectorization
>
> 15/12/03 16:18:54 [main]: DEBUG parse.TaskCompiler: Skipping stage id
> rearranger
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG
> method=SparkOptimizeTaskTree start=1449159534495 end=1449159534510
> duration=15 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
>
> 15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Completed plan
> generation
>
> 15/12/03 16:18:54 [main]: INFO ql.Driver: Semantic Analysis Completed
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: validation start
>
> 15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: not validating
> writeEntity, because entity is neither table nor partition
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG
> method=semanticAnalyze start=1449159533916 end=1449159534511 duration=595
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with:
> columnNames=[_col0] columnTypes=[bigint] separator=[[B@710bed9c]
> nullstring=\N lastColumnTakesRest=false timestampFormats=null
>
> 15/12/03 16:18:54 [main]: INFO exec.ListSinkOperator: Initializing
> operator OP[7]
>
> 15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe:
> org.apache.hadoop.hive.serde2.DelimitedJSONSerDe initialized with:
> columnNames=[] columnTypes=[] separator=[[B@7ead8de4] nullstring=NULL
> lastColumnTakesRest=false timestampFormats=null
>
> 15/12/03 16:18:54 [main]: INFO exec.ListSinkOperator: Initialization Done
> 7 OP
>
> 15/12/03 16:18:54 [main]: INFO exec.ListSinkOperator: Operator 7 OP
> initialized
>
> 15/12/03 16:18:54 [main]: INFO ql.Driver: Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)],
> properties:null)
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=compile
> start=1449159533903 end=1449159534532 duration=629
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:54 [main]: INFO metadata.Hive: Dumping metastore api call
> timing information for : compilation phase
>
> 15/12/03 16:18:54 [main]: DEBUG metadata.Hive: Total time spent in each
> metastore function (ms): {isCompatibleWith_(HiveConf, )=1,
> getTable_(String, String, )=37}
>
> 15/12/03 16:18:54 [main]: INFO ql.Driver: Concurrency mode is disabled,
> not creating a lock manager
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG
> method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:54 [main]: INFO ql.Driver: Starting
> command(queryId=hduser_20151203161853_0e621371-6907-430e-856a-998d32d0c314):
> select count(1) from t
>
> Query ID = hduser_20151203161853_0e621371-6907-430e-856a-998d32d0c314
>
> 15/12/03 16:18:54 [main]: INFO ql.Driver: Query ID =
> hduser_20151203161853_0e621371-6907-430e-856a-998d32d0c314
>
> Total jobs = 1
>
> 15/12/03 16:18:54 [main]: INFO ql.Driver: Total jobs = 1
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG
> method=TimeToSubmit start=1449159533903 end=1449159534532 duration=629
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=runTasks
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG
> method=task.SPARK.Stage-1 from=org.apache.hadoop.hive.ql.Driver>
>
> Launching Job 1 out of 1
>
> 15/12/03 16:18:54 [main]: INFO ql.Driver: Launching Job 1 out of 1
>
> 15/12/03 16:18:54 [main]: INFO ql.Driver: Starting task [Stage-1:MAPRED]
> in serial mode
>
> In order to change the average load for a reducer (in bytes):
>
> 15/12/03 16:18:54 [main]: INFO spark.SparkTask: In order to change the
> average load for a reducer (in bytes):
>
> set hive.exec.reducers.bytes.per.reducer=<number>
>
> 15/12/03 16:18:54 [main]: INFO spark.SparkTask: set
> hive.exec.reducers.bytes.per.reducer=<number>
>
> In order to limit the maximum number of reducers:
>
> 15/12/03 16:18:54 [main]: INFO spark.SparkTask: In order to limit the
> maximum number of reducers:
>
> set hive.exec.reducers.max=<number>
>
> 15/12/03 16:18:54 [main]: INFO spark.SparkTask: set
> hive.exec.reducers.max=<number>
>
> In order to set a constant number of reducers:
>
> 15/12/03 16:18:54 [main]: INFO spark.SparkTask: In order to set a constant
> number of reducers:
>
> set mapreduce.job.reduces=<number>
>
> 15/12/03 16:18:54 [main]: INFO spark.SparkTask: set
> mapreduce.job.reduces=<number>
>
> 15/12/03 16:18:54 [main]: INFO session.SparkSessionManagerImpl: Setting up
> the session manager.
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.connect.timeout ->
> 1000).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.serializer ->
> org.apache.spark.serializer.KryoSerializer).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.eventLog.enabled -> true).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.eventLog.dir ->
> /usr/lib/spark/logs).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.rpc.threads -> 8).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.secret.bits -> 256).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.home -> /usr/lib/spark).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.rpc.max.size ->
> 52428800).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.master -> spark://rhes564:7077).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.executor.memory -> 512m).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.server.connect.timeout
> -> 90000).
>
> 15/12/03 16:18:54 [main]: DEBUG logging.InternalLoggerFactory: Using SLF4J
> as the default logging framework
>
> 15/12/03 16:18:54 [main]: DEBUG channel.MultithreadEventLoopGroup:
> -Dio.netty.eventLoopThreads: 24
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0:
> java.nio.Buffer.address: available
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0:
> sun.misc.Unsafe.theUnsafe: available
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0:
> sun.misc.Unsafe.copyMemory: available
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0:
> java.nio.Bits.unaligned: true
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: UID: 1009
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: Java version: 7
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent:
> -Dio.netty.noUnsafe: false
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent:
> sun.misc.Unsafe: available
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent:
> -Dio.netty.noJavassist: false
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: Javassist:
> unavailable
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: You don't have
> Javassist in your class path or you don't have enough permission to load
> dynamically generated classes. Please check the configuration for better
> performance.
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent:
> -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent:
> -Dio.netty.bitMode: 64 (sun.arch.data.model)
>
> 15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent:
> -Dio.netty.noPreferDirect: false
>
> 15/12/03 16:18:54 [main]: DEBUG nio.NioEventLoop:
> -Dio.netty.noKeySetOptimization: false
>
> 15/12/03 16:18:54 [main]: DEBUG nio.NioEventLoop:
> -Dio.netty.selectorAutoRebuildThreshold: 512
>
> 15/12/03 16:18:54 [main]: DEBUG internal.ThreadLocalRandom:
> -Dio.netty.initialSeedUniquifier: 0x0b4453d9b0adc2b0 (took 0 ms)
>
> 15/12/03 16:18:54 [main]: DEBUG buffer.ByteBufUtil:
> -Dio.netty.allocator.type: unpooled
>
> 15/12/03 16:18:54 [main]: DEBUG buffer.ByteBufUtil:
> -Dio.netty.threadLocalDirectBufferSize: 65536
>
> 15/12/03 16:18:54 [main]: DEBUG util.NetUtil: Loopback interface: lo (lo,
> 127.0.0.1)
>
> 15/12/03 16:18:54 [main]: DEBUG util.NetUtil:
> /proc/sys/net/core/somaxconn: 128
>
> 15/12/03 16:18:54 [main]: WARN rpc.RpcConfiguration: Your hostname,
> rhes564, resolves to a loopback address, but we couldn't find any external
> IP address!
>
> 15/12/03 16:18:54 [main]: WARN rpc.RpcConfiguration: Set
> hive.spark.client.server.address if you need to bind to another address.
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.connect.timeout ->
> 1000).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.serializer ->
> org.apache.spark.serializer.KryoSerializer).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.eventLog.enabled -> true).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.eventLog.dir ->
> /usr/lib/spark/logs).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.rpc.threads -> 8).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.secret.bits -> 256).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.home -> /usr/lib/spark).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.rpc.max.size ->
> 52428800).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.master -> spark://rhes564:7077).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark
> property from hive configuration (spark.executor.memory -> 512m).
>
> 15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC
> property from hive configuration (hive.spark.client.server.connect.timeout
> -> 90000).
>
> 15/12/03 16:18:54 [main]: INFO client.SparkClientImpl: Running client
> driver with argv: /usr/lib/spark/bin/spark-submit --properties-file
> /tmp/spark-submit.4668566717444853712.properties --class *org.apache.hive.spark.client.RemoteDriver
> /usr/lib/hive/lib/hive-exec-1.2.1.jar --remote-host rhes564 --remote-port
> 36561 --conf hive.spark.client.connect.timeout=1000 --conf
> hive.spark.client.server.connect.timeout=90000 --conf
> hive.spark.client.channel.log.level=null --conf
> hive.spark.client.rpc.max.size=52428800 --conf
> hive.spark.client.rpc.threads=8 --conf hive.spark.client.secret.bits=256*
>
> 15/12/03 16:18:55 [stderr-redir-1]: INFO client.SparkClientImpl: Spark
> assembly has been built with Hive, including Datanucleus jars on classpath
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property: hive.spark.client.connect.timeout=1000
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property: hive.spark.client.rpc.threads=8
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property: hive.spark.client.rpc.max.size=52428800
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property: hive.spark.client.secret.bits=256
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property:
> hive.spark.client.server.connect.timeout=90000
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/03
> 16:18:56 INFO client.RemoteDriver: Connecting to: rhes564:36561
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Exception
> in thread "main" java.lang.NoSuchFieldError:
> SPARK_RPC_CLIENT_CONNECT_TIMEOUT
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:46)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:146)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> java.lang.reflect.Method.invoke(Method.java:606)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>
> 15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> 15/12/03 16:18:57 [Driver]: WARN client.SparkClientImpl: Child process
> exited with code 1.
>
> 15/12/03 16:19:04 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser: closed
>
> 15/12/03 16:19:04 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser:
> stopped, remaining connections 0
>
> 15/12/03 16:20:24 [main]: WARN client.SparkClientImpl: Error while waiting
> for client to connect.
>
> java.util.concurrent.ExecutionException:
> java.util.concurrent.TimeoutException: Timed out waiting for client
> connection.
>
> at
> io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
>
> at
> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
>
> at
> org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
>
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
>
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
>
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Caused by: java.util.concurrent.TimeoutException: Timed out waiting for
> client connection.
>
> at
> org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
>
> at
> io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
>
> at
> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
>
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
>
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
>
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>
> at java.lang.Thread.run(Thread.java:724)
>
> Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
> client.)'
>
> 15/12/03 16:20:24 [main]: ERROR spark.SparkTask: Failed to execute spark
> task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
> client.)'
>
> org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark
> client.
>
> at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
>
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
>
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
>
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> java.util.concurrent.TimeoutException: Timed out waiting for client
> connection.
>
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
>
> at
> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:109)
>
> at
> org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
>
> ... 21 more
>
> Caused by: java.util.concurrent.ExecutionException:
> java.util.concurrent.TimeoutException: Timed out waiting for client
> connection.
>
> at
> io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
>
> at
> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
>
> ... 25 more
>
> Caused by: java.util.concurrent.TimeoutException: Timed out waiting for
> client connection.
>
> at
> org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
>
> at
> io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
>
> at
> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
>
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
>
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
>
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>
> at java.lang.Thread.run(Thread.java:724)
>
>
>
> 15/12/03 16:20:24 [main]: ERROR spark.SparkTask: Failed to execute spark
> task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
> client.)'
>
> org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark
> client.
>
> at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
>
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
>
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
>
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> java.util.concurrent.TimeoutException: Timed out waiting for client
> connection.
>
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
>
> at
> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:109)
>
> at
> org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
>
> at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
>
> ... 21 more
>
> Caused by: java.util.concurrent.ExecutionException:
> java.util.concurrent.TimeoutException: Timed out waiting for client
> connection.
>
> at
> io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
>
> at
> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
>
> ... 25 more
>
> Caused by: java.util.concurrent.TimeoutException: Timed out waiting for
> client connection.
>
> at
> org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
>
> at
> io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
>
> at
> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
>
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
>
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
>
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>
> at java.lang.Thread.run(Thread.java:724)
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
>
> 15/12/03 16:20:24 [main]: ERROR ql.Driver: FAILED: Execution Error, return
> code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
>
> 15/12/03 16:20:24 [main]: DEBUG ql.Driver: Shutting down query select
> count(1) from t
>
> 15/12/03 16:20:24 [main]: INFO metadata.Hive: Dumping metastore api call
> timing information for : execution phase
>
> 15/12/03 16:20:24 [main]: DEBUG metadata.Hive: Total time spent in each
> metastore function (ms): {isCompatibleWith_(HiveConf, )=0}
>
> 15/12/03 16:20:24 [main]: INFO log.PerfLogger: </PERFLOG
> method=Driver.execute start=1449159534532 end=1449159624872 duration=90340
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:20:24 [main]: INFO log.PerfLogger: <PERFLOG
> method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:20:24 [main]: INFO log.PerfLogger: </PERFLOG
> method=releaseLocks start=1449159624872 end=1449159624872 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:20:24 [main]: INFO exec.ListSinkOperator: 7 finished.
> closing...
>
> 15/12/03 16:20:24 [main]: INFO exec.ListSinkOperator: 7 Close done
>
> 15/12/03 16:20:24 [main]: DEBUG ql.Driver: Shutting down query select
> count(1) from t
>
> 15/12/03 16:20:24 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
>
> 15/12/03 16:20:24 [main]: DEBUG ipc.Client: Connecting to rhes564/
> 50.140.197.217:9000
>
> 15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser:
> starting, having connections 1
>
> 15/12/03 16:20:24 [IPC Parameter Sending Thread #1]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #28
>
> 15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #28
>
> 15/12/03 16:20:24 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took
> 7ms
>
> 15/12/03 16:20:24 [IPC Parameter Sending Thread #1]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #29
>
> 15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #29
>
> 15/12/03 16:20:24 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took
> 1ms
>
> 15/12/03 16:20:24 [IPC Parameter Sending Thread #1]: DEBUG ipc.Client: IPC
> Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser
> sending #30
>
> 15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser got
> value #30
>
> 15/12/03 16:20:24 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took
> 2ms
>
> 15/12/03 16:20:24 [main]: INFO log.PerfLogger: <PERFLOG
> method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
>
> 15/12/03 16:20:24 [main]: INFO log.PerfLogger: </PERFLOG
> method=releaseLocks start=1449159624886 end=1449159624886 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
>
> hive>
>
> >
>
> >
>
> >
>
> >
>
> > 15/12/03 16:20:34 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser: closed
>
> 15/12/03 16:20:34 [IPC Client (533248865) connection to rhes564/
> 50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client
> (533248865) connection to rhes564/50.140.197.217:9000 from hduser:
> stopped, remaining connections 0
>
>
>
> Mich Talebzadeh
>
>
>
> *Sybase ASE 15 Gold Medal Award 2008*
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
>
>
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE
> 15", ISBN 978-0-9563693-0-7*.
>
> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4*
>
> *Publications due shortly:*
>
> *Complex Event Processing in Heterogeneous Environments*, ISBN:
> 978-0-9563693-3-8
>
> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume
> one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Ltd, its subsidiaries nor their employees
> accept any responsibility.
>
>
>
> *From:* Xuefu Zhang [mailto:xzhang@cloudera.com]
> *Sent:* 03 December 2015 14:02
> *To:* user@hive.apache.org
> *Cc:* user@spark.apache.org
> *Subject:* Re: Building spark 1.3 from source code to work with Hive 1.2.1
>
>
>
> Mich,
>
> To start your Spark standalone cluster, you can just download the tarball
> from Spark repo site. In other words, you don't need to start your cluster
> using your build.
>
> You only need to spark-assembly.jar to Hive's /lib directory and that's it.
>
> I guess you have been confused by this, which I tried to explain
> previously.
>
> Thanks,
>
> Xuefu
>
>
>
>
>
> On Thu, Dec 3, 2015 at 2:28 AM, Mich Talebzadeh <mi...@peridale.co.uk>
> wrote:
>
> Hi,
>
>
>
> I have seen mails that state that the user has managed to build spark 1.3
> to work with Hive. I tried Spark 1.5.2 but no luck
>
>
>
> I downloaded spark source 1.3 source code spark-1.3.0.tar and built it as
> follows
>
>
>
> ./make-distribution.sh --name "hadoop2-without-hive" --tgz
> "-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"
>
>
>
> This successfully completed and created the tarred zip file. I then
> created spark 1.3 tree from this zipped file. $SPARK_HOME is /
> usr/lib/spark
>
>
>
> Other steps that I performed:
>
>
>
> 1. In $HIVE_HOME/lib , I copied spark-assembly-1.3.0-hadoop2.4.0.jar to
> this directory
>
> 2. In $SPARK_HOME/conf I created a syblink to
> /usr/lib/hive/conf/hive-site.xml
>
>
>
> Then I tried to start spark master node
>
>
>
> /usr/lib/spark/sbin/start-master.sh
>
>
>
> I get the following error:
>
>
>
>
>
> cat
> /usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out
>
> Spark Command: /usr/java/latest/bin/java -cp
> :/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/home/hduser/hadoop-2.6.0/etc/hadoop
> -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m
> org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port
> 8080
>
> ========================================
>
>
>
> Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
>
> at java.lang.Class.getDeclaredMethods0(Native Method)
>
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
>
> at java.lang.Class.getMethod0(Class.java:2764)
>
> at java.lang.Class.getMethod(Class.java:1653)
>
> at
> sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
>
> at
> sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
>
> Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>
>
> I also notice that in /usr/lib/spark/lib, I only have the following jar
> files
>
>
>
> -rw-r--r-- 1 hduser hadoop 98795479 Dec 3 09:03
> spark-examples-1.3.0-hadoop2.4.0.jar
>
> -rw-r--r-- 1 hduser hadoop 98187168 Dec 3 09:03
> spark-assembly-1.3.0-hadoop2.4.0.jar
>
> -rw-r--r-- 1 hduser hadoop 4136760 Dec 3 09:03
> spark-1.3.0-yarn-shuffle.jar
>
>
>
> Wheras in pre-build downloaded one à /usr/lib/spark-1.3.0-bin-hadoop2.4, there
> are additional JAR files
>
>
>
> -rw-rw-r-- 1 hduser hadoop 1890075 Mar 6 2015
> datanucleus-core-3.2.10.jar
>
> -rw-rw-r-- 1 hduser hadoop 112446389 Mar 6 2015
> spark-examples-1.3.0-hadoop2.4.0.jar
>
> -rw-rw-r-- 1 hduser hadoop 159319006 Mar 6 2015
> spark-assembly-1.3.0-hadoop2.4.0.jar
>
> -rw-rw-r-- 1 hduser hadoop 4136744 Mar 6 2015
> spark-1.3.0-yarn-shuffle.jar
>
> -rw-rw-r-- 1 hduser hadoop 1809447 Mar 6 2015
> datanucleus-rdbms-3.2.9.jar
>
> -rw-rw-r-- 1 hduser hadoop 339666 Mar 6 2015
> datanucleus-api-jdo-3.2.6.jar
>
>
>
> Any ideas what is is missing? I am sure someone has sorted this one out
> before.
>
>
>
>
>
> Thanks,
>
>
>
> Mich
>
>
>
>
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Ltd, its subsidiaries nor their employees
> accept any responsibility.
>
>
>
>
>
RE: Building spark 1.3 from source code to work with Hive 1.2.1
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Thanks.
I have done this before but little joy I am afraid and again errors!! Very disappointing
1. Downloaded pre-build spark-13.0 and installed it.
2. Went to $HIVE_HOME/lib directory and created a symlink to $SPARK_HOME/lib/spark-assembly-1.3.0-hadoop2.4.0.jar
3. Went to $SPARK_HOME/conf directory and created a symlink to $HIVE_HOME/conf/hive-site.xml
4. Started spark-master
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Spark Command: /usr/java/latest/bin/java -cp :/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/usr/lib/spark/lib/datanucleus-core-3.2.10.jar:/usr/lib/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/lib/spark/lib/datanucleus-rdbms-3.2.9.jar:/home/hduser/hadoop-2.6.0/etc/hadoop -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port 8080
========================================
15/12/03 15:35:53 INFO master.Master: Registered signal handlers for [TERM, HUP, INT]
15/12/03 15:35:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/12/03 15:35:54 INFO spark.SecurityManager: Changing view acls to: hduser
15/12/03 15:35:54 INFO spark.SecurityManager: Changing modify acls to: hduser
15/12/03 15:35:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser)
15/12/03 15:35:54 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/12/03 15:35:54 INFO Remoting: Starting remoting
15/12/03 15:35:54 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@rhes564:7077]
15/12/03 15:35:54 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkMaster@rhes564:7077]
15/12/03 15:35:54 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
15/12/03 15:35:55 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/03 15:35:55 INFO server.AbstractConnector: Started SelectChannelConnector@rhes564:6066
15/12/03 15:35:55 INFO util.Utils: Successfully started service on port 6066.
15/12/03 15:35:55 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
15/12/03 15:35:55 INFO master.Master: Starting Spark master at spark://rhes564:7077
15/12/03 15:35:55 INFO master.Master: Running Spark version 1.3.0
15/12/03 15:35:55 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/03 15:35:55 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080
15/12/03 15:35:55 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.
15/12/03 15:35:55 INFO ui.MasterWebUI: Started MasterWebUI at http://rhes564:8080
15/12/03 15:35:55 INFO master.Master: I have been elected leader! New state: ALIVE
15/12/03 15:36:58 INFO master.Master: Registering worker rhes564:56498 with 12 cores, 22.5 GB RAM
5. Started a worker process as start-slave.sh 1 spark://rhes564:7077
6. Went back to Hive in debug mode and did
set spark.home=/usr/lib/spark;
set hive.execution.engine=spark;
set spark.master=spark://rhes564:7077;
set spark.eventLog.enabled=true;
set spark.eventLog.dir=/usr/lib/spark/logs;
set spark.executor.memory=512m;
set spark.serializer=org.apache.spark.serializer.KryoSerializer;
set spark.home=/usr/lib/spark;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: /usr/lib/spark
hive> set hive.execution.engine=spark;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: spark
hive> set spark.master=spark://rhes564:7077;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: spark://rhes564:7077
hive> set spark.eventLog.enabled=true;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: true
hive> set spark.eventLog.dir=/usr/lib/spark/logs;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: /usr/lib/spark/logs
hive> set spark.executor.memory=512m;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: 512m
hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: org.apache.spark.serializer.KryoSerializer
7. select count(1) from t;
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: DEBUG parse.VariableSubstitution: Substitution is on: select count(1) from t
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: INFO parse.ParseDriver: Parsing command: select count(1) from t
15/12/03 16:18:53 [main]: INFO parse.ParseDriver: Parse Completed
15/12/03 16:18:53 [main]: INFO log.PerfLogger: </PERFLOG method=parse start=1449159533904 end=1449159533916 duration=12 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: DEBUG ql.Driver: Encoding valid txns info 9223372036854775807:
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Starting Semantic Analysis
15/12/03 16:18:53 [main]: DEBUG exec.FunctionRegistry: Looking up GenericUDAF: count
15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Completed phase 1 of Semantic Analysis
15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Get metadata for source tables
15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Get metadata for subqueries
15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Get metadata for destination tables
15/12/03 16:18:54 [main]: DEBUG hdfs.DFSClient: /tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1: masked=rwx------
15/12/03 16:18:54 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
15/12/03 16:18:54 [main]: DEBUG ipc.Client: Connecting to rhes564/50.140.197.217:9000
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: starting, having connections 1
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #9
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #9
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: mkdirs took 6ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #10
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #10
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [main]: INFO ql.Context: New scratch dir is hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1
15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Completed getting MetaData in Semantic Analysis
15/12/03 16:18:54 [main]: INFO parse.BaseSemanticAnalyzer: Not invoking CBO because the statement has too few joins
15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30) owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id, i64 data_object_id, varchar(19) object_type, timestamp created, timestamp last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1) temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace, varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2, varchar(32) attribute, i32 op_type, timestamp op_time}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Table Plan for t TS[0]
15/12/03 16:18:54 [main]: DEBUG exec.FunctionRegistry: Looking up GenericUDAF: count
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: RR before GB t{(owner,owner: varchar(30))(object_name,object_name: varchar(30))(subobject_name,subobject_name: varchar(30))(object_id,object_id: bigint)(data_object_id,data_object_id: bigint)(object_type,object_type: varchar(19))(created,created: timestamp)(last_ddl_time,last_ddl_time: timestamp)(timestamp,timestamp: varchar(19))(status,status: varchar(7))(temporary2,temporary2: varchar(1))(generated,generated: varchar(1))(secondary,secondary: varchar(1))(namespace,namespace: bigint)(edition_name,edition_name: varchar(30))(padding1,padding1: varchar(4000))(padding2,padding2: varchar(3500))(attribute,attribute: varchar(32))(op_type,op_type: int)(op_time,op_time: timestamp)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)(row__id,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>)} after GB {((tok_function count 1),_col0: bigint)}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: tree: (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION count 1)))
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: genSelectPlan: input = {((tok_function count 1),_col0: bigint)} starRr = t{(owner,owner: varchar(30))(object_name,object_name: varchar(30))(subobject_name,subobject_name: varchar(30))(object_id,object_id: bigint)(data_object_id,data_object_id: bigint)(object_type,object_type: varchar(19))(created,created: timestamp)(last_ddl_time,last_ddl_time: timestamp)(timestamp,timestamp: varchar(19))(status,status: varchar(7))(temporary2,temporary2: varchar(1))(generated,generated: varchar(1))(secondary,secondary: varchar(1))(namespace,namespace: bigint)(edition_name,edition_name: varchar(30))(padding1,padding1: varchar(4000))(padding2,padding2: varchar(3500))(attribute,attribute: varchar(32))(op_type,op_type: int)(op_time,op_time: timestamp)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)(row__id,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>)}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Select Plan row schema: null{(_c0,_col0: bigint)}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Select Plan for clause: insclause-0
15/12/03 16:18:54 [main]: DEBUG ql.Context: Created staging dir = hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1 for path = hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000
15/12/03 16:18:54 [main]: INFO common.FileUtils: Creating directory if it doesn't exist: hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #11
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #11
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #12
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #12
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #13
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #13
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #14
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #14
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [main]: DEBUG hdfs.DFSClient: /tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1: masked=rwxr-xr-x
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #15
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #15
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: mkdirs took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #16
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #16
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: {-chgrp,-R,supergroup,hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000}
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #17
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #17
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #18
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #18
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #19
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #19
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 1ms
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: Return value is :0
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: {-chmod,-R,700,hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000}
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #20
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #20
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #21
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #21
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: setPermission took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #22
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #22
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #23
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #23
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: setPermission took 5ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #24
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #24
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 0ms
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: Return value is :0
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #25
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #25
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: FileStatus{path=hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000; isDirectory=true; modification_time=1449159534181; access_time=0; owner=hduser; group=supergroup; permission=rwx------; isSymlink=false}
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #26
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #26
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[_col0] columnTypes=[bigint] separator=[[B@51669ea5] nullstring=\N lastColumnTakesRest=false timestampFormats=null
15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[_col0] columnTypes=[bigint] separator=[[B@15a2b981] nullstring=\N lastColumnTakesRest=false timestampFormats=null
15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Set stats collection dir : hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1/-ext-10002
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created FileSink Plan for clause: insclause-0dest_path: hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000 row schema: null{(_c0,_col0: bigint)}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Body Plan for Query Block null
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Plan for Query Block null
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Before logical optimization
TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-SEL[5]-FS[6]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:TS[0] with rs:(owner: varchar(30)|{t}owner,object_name: varchar(30)|{t}object_name,subobject_name: varchar(30)|{t}subobject_name,object_id: bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type: varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time: timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status: varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated: varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace: bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1: varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute: varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time: timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE: bigint|{t}block__offset__inside__file,INPUT__FILE__NAME: string|{t}input__file__name,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator TS[0]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:SEL[1] with rs:(owner: varchar(30)|{t}owner,object_name: varchar(30)|{t}object_name,subobject_name: varchar(30)|{t}subobject_name,object_id: bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type: varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time: timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status: varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated: varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace: bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1: varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute: varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time: timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE: bigint|{t}block__offset__inside__file,INPUT__FILE__NAME: string|{t}input__file__name,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator SEL[1]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory: New column list:(Column[owner] Column[object_name] Column[subobject_name] Column[object_id] Column[data_object_id] Column[object_type] Column[created] Column[last_ddl_time] Column[timestamp] Column[status] Column[temporary2] Column[generated] Column[secondary] Column[namespace] Column[edition_name] Column[padding1] Column[padding2] Column[attribute] Column[op_type] Column[op_time] Column[BLOCK__OFFSET__INSIDE__FILE] Column[INPUT__FILE__NAME] Column[ROW__ID])
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:GBY[2] with rs:(_col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator GBY[2]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:RS[3] with rs:(VALUE._col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator RS[3]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:GBY[4] with rs:(_col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator GBY[4]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:SEL[5] with rs:(_col0: bigint|{null}_c0)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator SEL[5]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory: New column list:(Column[_col0])
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:FS[6] with rs:(_col0: bigint|{})
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator FS[6]
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for FS(6)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for SEL(5)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for GBY(4)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for RS(3)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for GBY(2)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for SEL(1)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for TS(0)
15/12/03 16:18:54 [main]: DEBUG ppd.PredicatePushDown: After PPD:
TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-SEL[5]-FS[6]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:TS[0] with rs:(owner: varchar(30)|{t}owner,object_name: varchar(30)|{t}object_name,subobject_name: varchar(30)|{t}subobject_name,object_id: bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type: varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time: timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status: varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated: varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace: bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1: varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute: varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time: timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE: bigint|{t}block__offset__inside__file,INPUT__FILE__NAME: string|{t}input__file__name,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator TS[0]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:SEL[1] with rs:(owner: varchar(30)|{t}owner,object_name: varchar(30)|{t}object_name,subobject_name: varchar(30)|{t}subobject_name,object_id: bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type: varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time: timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status: varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated: varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace: bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1: varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute: varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time: timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE: bigint|{t}block__offset__inside__file,INPUT__FILE__NAME: string|{t}input__file__name,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator SEL[1]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory: New column list:(Column[owner] Column[object_name] Column[subobject_name] Column[object_id] Column[data_object_id] Column[object_type] Column[created] Column[last_ddl_time] Column[timestamp] Column[status] Column[temporary2] Column[generated] Column[secondary] Column[namespace] Column[edition_name] Column[padding1] Column[padding2] Column[attribute] Column[op_type] Column[op_time] Column[BLOCK__OFFSET__INSIDE__FILE] Column[INPUT__FILE__NAME] Column[ROW__ID])
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:GBY[2] with rs:(_col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator GBY[2]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:RS[3] with rs:(VALUE._col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator RS[3]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:GBY[4] with rs:(_col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator GBY[4]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:SEL[5] with rs:(_col0: bigint|{null}_c0)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator SEL[5]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory: New column list:(Column[_col0])
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:FS[6] with rs:(_col0: bigint|{})
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator FS[6]
15/12/03 16:18:54 [main]: DEBUG optimizer.ColumnPrunerProcFactory: Reduce Sink Operator 3 key:[]
15/12/03 16:18:54 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 3 oldColExprMap: {VALUE._col0=Column[_col0]}
15/12/03 16:18:54 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 3 newColExprMap: {VALUE._col0=Column[_col0]}
15/12/03 16:18:54 [main]: DEBUG optimizer.IdentityProjectRemover: Identity project remover optimization removed : SEL[5]
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=partition-retrieving from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=partition-retrieving start=1449159534437 end=1449159534440 duration=3 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
15/12/03 16:18:54 [main]: DEBUG exec.TableScanOperator: Setting stats (Num rows: 1767886 Data size: 5984968162 Basic stats: COMPLETE Column stats: COMPLETE) on TS[0]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-TS[0] (t): numRows: 1767886 dataSize: 5984968162 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {}
15/12/03 16:18:54 [main]: DEBUG exec.SelectOperator: Setting stats (Num rows: 1767886 Data size: 5984968162 Basic stats: COMPLETE Column stats: COMPLETE) on SEL[1]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-SEL[1]: numRows: 1767886 dataSize: 5984968162 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {}
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: STATS-GBY[2]: inputSize: 5984968162 maxSplitSize: 256000000 parallelism: 24 containsGroupingSet: false sizeOfGroupingSet: 1
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: STATS-GBY[2] hashAgg: true
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [Case 3] STATS-GBY[2]: cardinality: 24
15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting stats (Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE) on GBY[2]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-GBY[2]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {_col0= colName: _col0 colType: bigint countDistincts: 24 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false}
15/12/03 16:18:54 [main]: DEBUG exec.ReduceSinkOperator: Setting stats (Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE) on RS[3]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-RS[3]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {VALUE._col0= colName: VALUE._col0 colType: bigint countDistincts: 24 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false}
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: STATS-GBY[4]: inputSize: 1 maxSplitSize: 256000000 parallelism: 1 containsGroupingSet: false sizeOfGroupingSet: 1
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: STATS-GBY[4] hashAgg: false
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [Case 9] STATS-GBY[4]: cardinality: 1
15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting stats (Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE) on GBY[4]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-GBY[4]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {_col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false}
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-FS[6]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {_col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false}
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #27
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #27
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 1ms
15/12/03 16:18:54 [main]: DEBUG exec.TableScanOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@34dccb3d) on TS[0]
15/12/03 16:18:54 [main]: DEBUG exec.SelectOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@ba32ce8) on SEL[1]
15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@5d93950) on GBY[2]
15/12/03 16:18:54 [main]: DEBUG exec.ReduceSinkOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@7841b62f) on RS[3]
15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@22200557) on GBY[4]
15/12/03 16:18:54 [main]: DEBUG exec.FileSinkOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@22200557) on FS[6]
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: After logical optimization
TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-FS[6]
15/12/03 16:18:54 [main]: DEBUG security.Groups: Returning cached groups for 'hduser'
15/12/03 16:18:54 [main]: DEBUG security.Groups: Returning cached groups for 'hduser'
15/12/03 16:18:54 [main]: DEBUG session.SessionState: Session is using authorization class class org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=SparkOptimizeOperatorTree from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: INFO spark.SetSparkReducerParallelism: Number of reducers determined to be: 1
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=SparkOptimizeOperatorTree start=1449159534470 end=1449159534477 duration=7 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=SparkGenerateTaskTree from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Root operator: TS[0]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Leaf operator: RS[3]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkUtils: Adding map work (Map 1) for TS[0]
15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30) owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id, i64 data_object_id, varchar(19) object_type, timestamp created, timestamp last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1) temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace, varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2, varchar(32) attribute, i32 op_type, timestamp op_time}
15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30) owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id, i64 data_object_id, varchar(19) object_type, timestamp created, timestamp last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1) temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace, varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2, varchar(32) attribute, i32 op_type, timestamp op_time}
15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30) owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id, i64 data_object_id, varchar(19) object_type, timestamp created, timestamp last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1) temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace, varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2, varchar(32) attribute, i32 op_type, timestamp op_time}
15/12/03 16:18:54 [main]: DEBUG optimizer.GenMapRedUtils: Adding hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t of tablet
15/12/03 16:18:54 [main]: DEBUG optimizer.GenMapRedUtils: Information added for path hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: First pass. Leaf operator: RS[3]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Root operator: GBY[4]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Leaf operator: FS[6]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkUtils: Adding reduce work (Reducer 2) for GBY[4]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkUtils: Setting up reduce sink: RS[3] with following reduce work: Reducer 2
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Removing RS[3] as parent from GBY[4]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: First pass. Leaf operator: FS[6]
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=SparkGenerateTaskTree start=1449159534477 end=1449159534495 duration=18 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=SparkOptimizeTaskTree from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: DEBUG parse.TaskCompiler: Skipping runtime skew join optimization
15/12/03 16:18:54 [main]: DEBUG physical.NullScanTaskDispatcher: Looking at: Map 1
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
15/12/03 16:18:54 [main]: DEBUG physical.NullScanTaskDispatcher: Looking at: Map 1
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
15/12/03 16:18:54 [main]: DEBUG physical.NullScanTaskDispatcher: Looking at: Map 1
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
15/12/03 16:18:54 [main]: DEBUG parse.TaskCompiler: Skipping vectorization
15/12/03 16:18:54 [main]: DEBUG parse.TaskCompiler: Skipping stage id rearranger
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=SparkOptimizeTaskTree start=1449159534495 end=1449159534510 duration=15 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Completed plan generation
15/12/03 16:18:54 [main]: INFO ql.Driver: Semantic Analysis Completed
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: validation start
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: not validating writeEntity, because entity is neither table nor partition
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=semanticAnalyze start=1449159533916 end=1449159534511 duration=595 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[_col0] columnTypes=[bigint] separator=[[B@710bed9c] nullstring=\N lastColumnTakesRest=false timestampFormats=null
15/12/03 16:18:54 [main]: INFO exec.ListSinkOperator: Initializing operator OP[7]
15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.DelimitedJSONSerDe initialized with: columnNames=[] columnTypes=[] separator=[[B@7ead8de4] nullstring=NULL lastColumnTakesRest=false timestampFormats=null
15/12/03 16:18:54 [main]: INFO exec.ListSinkOperator: Initialization Done 7 OP
15/12/03 16:18:54 [main]: INFO exec.ListSinkOperator: Operator 7 OP initialized
15/12/03 16:18:54 [main]: INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=compile start=1449159533903 end=1449159534532 duration=629 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: INFO metadata.Hive: Dumping metastore api call timing information for : compilation phase
15/12/03 16:18:54 [main]: DEBUG metadata.Hive: Total time spent in each metastore function (ms): {isCompatibleWith_(HiveConf, )=1, getTable_(String, String, )=37}
15/12/03 16:18:54 [main]: INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: INFO ql.Driver: Starting command(queryId=hduser_20151203161853_0e621371-6907-430e-856a-998d32d0c314): select count(1) from t
Query ID = hduser_20151203161853_0e621371-6907-430e-856a-998d32d0c314
15/12/03 16:18:54 [main]: INFO ql.Driver: Query ID = hduser_20151203161853_0e621371-6907-430e-856a-998d32d0c314
Total jobs = 1
15/12/03 16:18:54 [main]: INFO ql.Driver: Total jobs = 1
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=TimeToSubmit start=1449159533903 end=1449159534532 duration=629 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=task.SPARK.Stage-1 from=org.apache.hadoop.hive.ql.Driver>
Launching Job 1 out of 1
15/12/03 16:18:54 [main]: INFO ql.Driver: Launching Job 1 out of 1
15/12/03 16:18:54 [main]: INFO ql.Driver: Starting task [Stage-1:MAPRED] in serial mode
In order to change the average load for a reducer (in bytes):
15/12/03 16:18:54 [main]: INFO spark.SparkTask: In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
15/12/03 16:18:54 [main]: INFO spark.SparkTask: set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
15/12/03 16:18:54 [main]: INFO spark.SparkTask: In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
15/12/03 16:18:54 [main]: INFO spark.SparkTask: set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
15/12/03 16:18:54 [main]: INFO spark.SparkTask: In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
15/12/03 16:18:54 [main]: INFO spark.SparkTask: set mapreduce.job.reduces=<number>
15/12/03 16:18:54 [main]: INFO session.SparkSessionManagerImpl: Setting up the session manager.
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.connect.timeout -> 1000).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.serializer -> org.apache.spark.serializer.KryoSerializer).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.eventLog.enabled -> true).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.eventLog.dir -> /usr/lib/spark/logs).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.rpc.threads -> 8).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.secret.bits -> 256).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.home -> /usr/lib/spark).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.rpc.max.size -> 52428800).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.master -> spark://rhes564:7077).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.executor.memory -> 512m).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.server.connect.timeout -> 90000).
15/12/03 16:18:54 [main]: DEBUG logging.InternalLoggerFactory: Using SLF4J as the default logging framework
15/12/03 16:18:54 [main]: DEBUG channel.MultithreadEventLoopGroup: -Dio.netty.eventLoopThreads: 24
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0: java.nio.Buffer.address: available
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0: sun.misc.Unsafe.theUnsafe: available
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0: sun.misc.Unsafe.copyMemory: available
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0: java.nio.Bits.unaligned: true
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: UID: 1009
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: Java version: 7
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.noUnsafe: false
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: sun.misc.Unsafe: available
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.noJavassist: false
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: Javassist: unavailable
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.bitMode: 64 (sun.arch.data.model)
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.noPreferDirect: false
15/12/03 16:18:54 [main]: DEBUG nio.NioEventLoop: -Dio.netty.noKeySetOptimization: false
15/12/03 16:18:54 [main]: DEBUG nio.NioEventLoop: -Dio.netty.selectorAutoRebuildThreshold: 512
15/12/03 16:18:54 [main]: DEBUG internal.ThreadLocalRandom: -Dio.netty.initialSeedUniquifier: 0x0b4453d9b0adc2b0 (took 0 ms)
15/12/03 16:18:54 [main]: DEBUG buffer.ByteBufUtil: -Dio.netty.allocator.type: unpooled
15/12/03 16:18:54 [main]: DEBUG buffer.ByteBufUtil: -Dio.netty.threadLocalDirectBufferSize: 65536
15/12/03 16:18:54 [main]: DEBUG util.NetUtil: Loopback interface: lo (lo, 127.0.0.1)
15/12/03 16:18:54 [main]: DEBUG util.NetUtil: /proc/sys/net/core/somaxconn: 128
15/12/03 16:18:54 [main]: WARN rpc.RpcConfiguration: Your hostname, rhes564, resolves to a loopback address, but we couldn't find any external IP address!
15/12/03 16:18:54 [main]: WARN rpc.RpcConfiguration: Set hive.spark.client.server.address if you need to bind to another address.
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.connect.timeout -> 1000).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.serializer -> org.apache.spark.serializer.KryoSerializer).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.eventLog.enabled -> true).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.eventLog.dir -> /usr/lib/spark/logs).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.rpc.threads -> 8).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.secret.bits -> 256).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.home -> /usr/lib/spark).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.rpc.max.size -> 52428800).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.master -> spark://rhes564:7077).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.executor.memory -> 512m).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.server.connect.timeout -> 90000).
15/12/03 16:18:54 [main]: INFO client.SparkClientImpl: Running client driver with argv: /usr/lib/spark/bin/spark-submit --properties-file /tmp/spark-submit.4668566717444853712.properties --class org.apache.hive.spark.client.RemoteDriver /usr/lib/hive/lib/hive-exec-1.2.1.jar --remote-host rhes564 --remote-port 36561 --conf hive.spark.client.connect.timeout=1000 --conf hive.spark.client.server.connect.timeout=90000 --conf hive.spark.client.channel.log.level=null --conf hive.spark.client.rpc.max.size=52428800 --conf hive.spark.client.rpc.threads=8 --conf hive.spark.client.secret.bits=256
15/12/03 16:18:55 [stderr-redir-1]: INFO client.SparkClientImpl: Spark assembly has been built with Hive, including Datanucleus jars on classpath
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.connect.timeout=1000
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.rpc.threads=8
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.rpc.max.size=52428800
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.secret.bits=256
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.server.connect.timeout=90000
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/03 16:18:56 INFO client.RemoteDriver: Connecting to: rhes564:36561
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_CLIENT_CONNECT_TIMEOUT
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:46)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:146)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at java.lang.reflect.Method.invoke(Method.java:606)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/12/03 16:18:57 [Driver]: WARN client.SparkClientImpl: Child process exited with code 1.
15/12/03 16:19:04 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: closed
15/12/03 16:19:04 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: stopped, remaining connections 0
15/12/03 16:20:24 [main]: WARN client.SparkClientImpl: Error while waiting for client to connect.
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:724)
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
15/12/03 16:20:24 [main]: ERROR spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:109)
at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
... 21 more
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
... 25 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:724)
15/12/03 16:20:24 [main]: ERROR spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:109)
at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
... 21 more
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
... 25 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:724)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
15/12/03 16:20:24 [main]: ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
15/12/03 16:20:24 [main]: DEBUG ql.Driver: Shutting down query select count(1) from t
15/12/03 16:20:24 [main]: INFO metadata.Hive: Dumping metastore api call timing information for : execution phase
15/12/03 16:20:24 [main]: DEBUG metadata.Hive: Total time spent in each metastore function (ms): {isCompatibleWith_(HiveConf, )=0}
15/12/03 16:20:24 [main]: INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1449159534532 end=1449159624872 duration=90340 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:20:24 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:20:24 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1449159624872 end=1449159624872 duration=0 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:20:24 [main]: INFO exec.ListSinkOperator: 7 finished. closing...
15/12/03 16:20:24 [main]: INFO exec.ListSinkOperator: 7 Close done
15/12/03 16:20:24 [main]: DEBUG ql.Driver: Shutting down query select count(1) from t
15/12/03 16:20:24 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
15/12/03 16:20:24 [main]: DEBUG ipc.Client: Connecting to rhes564/50.140.197.217:9000
15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: starting, having connections 1
15/12/03 16:20:24 [IPC Parameter Sending Thread #1]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #28
15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #28
15/12/03 16:20:24 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 7ms
15/12/03 16:20:24 [IPC Parameter Sending Thread #1]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #29
15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #29
15/12/03 16:20:24 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 1ms
15/12/03 16:20:24 [IPC Parameter Sending Thread #1]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #30
15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #30
15/12/03 16:20:24 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 2ms
15/12/03 16:20:24 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:20:24 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1449159624886 end=1449159624886 duration=0 from=org.apache.hadoop.hive.ql.Driver>
hive>
>
>
>
>
> 15/12/03 16:20:34 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: closed
15/12/03 16:20:34 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: stopped, remaining connections 0
Mich Talebzadeh
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Xuefu Zhang [mailto:xzhang@cloudera.com]
Sent: 03 December 2015 14:02
To: user@hive.apache.org
Cc: user@spark.apache.org
Subject: Re: Building spark 1.3 from source code to work with Hive 1.2.1
Mich,
To start your Spark standalone cluster, you can just download the tarball from Spark repo site. In other words, you don't need to start your cluster using your build.
You only need to spark-assembly.jar to Hive's /lib directory and that's it.
I guess you have been confused by this, which I tried to explain previously.
Thanks,
Xuefu
On Thu, Dec 3, 2015 at 2:28 AM, Mich Talebzadeh <mich@peridale.co.uk <ma...@peridale.co.uk> > wrote:
Hi,
I have seen mails that state that the user has managed to build spark 1.3 to work with Hive. I tried Spark 1.5.2 but no luck
I downloaded spark source 1.3 source code spark-1.3.0.tar and built it as follows
./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"
This successfully completed and created the tarred zip file. I then created spark 1.3 tree from this zipped file. $SPARK_HOME is /usr/lib/spark
Other steps that I performed:
1. In $HIVE_HOME/lib , I copied spark-assembly-1.3.0-hadoop2.4.0.jar to this directory
2. In $SPARK_HOME/conf I created a syblink to /usr/lib/hive/conf/hive-site.xml
Then I tried to start spark master node
/usr/lib/spark/sbin/start-master.sh
I get the following error:
cat /usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out
Spark Command: /usr/java/latest/bin/java -cp :/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/home/hduser/hadoop-2.6.0/etc/hadoop -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port 8080
========================================
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
at java.lang.Class.getMethod0(Class.java:2764)
at java.lang.Class.getMethod(Class.java:1653)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I also notice that in /usr/lib/spark/lib, I only have the following jar files
-rw-r--r-- 1 hduser hadoop 98795479 Dec 3 09:03 spark-examples-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 98187168 Dec 3 09:03 spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 4136760 Dec 3 09:03 spark-1.3.0-yarn-shuffle.jar
Wheras in pre-build downloaded one --> /usr/lib/spark-1.3.0-bin-hadoop2.4, there are additional JAR files
-rw-rw-r-- 1 hduser hadoop 1890075 Mar 6 2015 datanucleus-core-3.2.10.jar
-rw-rw-r-- 1 hduser hadoop 112446389 Mar 6 2015 spark-examples-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 159319006 Mar 6 2015 spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 4136744 Mar 6 2015 spark-1.3.0-yarn-shuffle.jar
-rw-rw-r-- 1 hduser hadoop 1809447 Mar 6 2015 datanucleus-rdbms-3.2.9.jar
-rw-rw-r-- 1 hduser hadoop 339666 Mar 6 2015 datanucleus-api-jdo-3.2.6.jar
Any ideas what is is missing? I am sure someone has sorted this one out before.
Thanks,
Mich
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
RE: Building spark 1.3 from source code to work with Hive 1.2.1
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Thanks.
I have done this before but little joy I am afraid and again errors!! Very disappointing
1. Downloaded pre-build spark-13.0 and installed it.
2. Went to $HIVE_HOME/lib directory and created a symlink to $SPARK_HOME/lib/spark-assembly-1.3.0-hadoop2.4.0.jar
3. Went to $SPARK_HOME/conf directory and created a symlink to $HIVE_HOME/conf/hive-site.xml
4. Started spark-master
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Spark Command: /usr/java/latest/bin/java -cp :/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/usr/lib/spark/lib/datanucleus-core-3.2.10.jar:/usr/lib/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/lib/spark/lib/datanucleus-rdbms-3.2.9.jar:/home/hduser/hadoop-2.6.0/etc/hadoop -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port 8080
========================================
15/12/03 15:35:53 INFO master.Master: Registered signal handlers for [TERM, HUP, INT]
15/12/03 15:35:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/12/03 15:35:54 INFO spark.SecurityManager: Changing view acls to: hduser
15/12/03 15:35:54 INFO spark.SecurityManager: Changing modify acls to: hduser
15/12/03 15:35:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser)
15/12/03 15:35:54 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/12/03 15:35:54 INFO Remoting: Starting remoting
15/12/03 15:35:54 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@rhes564:7077]
15/12/03 15:35:54 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkMaster@rhes564:7077]
15/12/03 15:35:54 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
15/12/03 15:35:55 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/03 15:35:55 INFO server.AbstractConnector: Started SelectChannelConnector@rhes564:6066
15/12/03 15:35:55 INFO util.Utils: Successfully started service on port 6066.
15/12/03 15:35:55 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
15/12/03 15:35:55 INFO master.Master: Starting Spark master at spark://rhes564:7077
15/12/03 15:35:55 INFO master.Master: Running Spark version 1.3.0
15/12/03 15:35:55 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/03 15:35:55 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080
15/12/03 15:35:55 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.
15/12/03 15:35:55 INFO ui.MasterWebUI: Started MasterWebUI at http://rhes564:8080
15/12/03 15:35:55 INFO master.Master: I have been elected leader! New state: ALIVE
15/12/03 15:36:58 INFO master.Master: Registering worker rhes564:56498 with 12 cores, 22.5 GB RAM
5. Started a worker process as start-slave.sh 1 spark://rhes564:7077
6. Went back to Hive in debug mode and did
set spark.home=/usr/lib/spark;
set hive.execution.engine=spark;
set spark.master=spark://rhes564:7077;
set spark.eventLog.enabled=true;
set spark.eventLog.dir=/usr/lib/spark/logs;
set spark.executor.memory=512m;
set spark.serializer=org.apache.spark.serializer.KryoSerializer;
set spark.home=/usr/lib/spark;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: /usr/lib/spark
hive> set hive.execution.engine=spark;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: spark
hive> set spark.master=spark://rhes564:7077;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: spark://rhes564:7077
hive> set spark.eventLog.enabled=true;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: true
hive> set spark.eventLog.dir=/usr/lib/spark/logs;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: /usr/lib/spark/logs
hive> set spark.executor.memory=512m;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: 512m
hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
15/12/03 16:09:06 [main]: DEBUG parse.VariableSubstitution: Substitution is on: org.apache.spark.serializer.KryoSerializer
7. select count(1) from t;
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: DEBUG parse.VariableSubstitution: Substitution is on: select count(1) from t
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: INFO parse.ParseDriver: Parsing command: select count(1) from t
15/12/03 16:18:53 [main]: INFO parse.ParseDriver: Parse Completed
15/12/03 16:18:53 [main]: INFO log.PerfLogger: </PERFLOG method=parse start=1449159533904 end=1449159533916 duration=12 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: DEBUG ql.Driver: Encoding valid txns info 9223372036854775807:
15/12/03 16:18:53 [main]: INFO log.PerfLogger: <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Starting Semantic Analysis
15/12/03 16:18:53 [main]: DEBUG exec.FunctionRegistry: Looking up GenericUDAF: count
15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Completed phase 1 of Semantic Analysis
15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Get metadata for source tables
15/12/03 16:18:53 [main]: INFO parse.CalcitePlanner: Get metadata for subqueries
15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Get metadata for destination tables
15/12/03 16:18:54 [main]: DEBUG hdfs.DFSClient: /tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1: masked=rwx------
15/12/03 16:18:54 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
15/12/03 16:18:54 [main]: DEBUG ipc.Client: Connecting to rhes564/50.140.197.217:9000
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: starting, having connections 1
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #9
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #9
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: mkdirs took 6ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #10
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #10
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [main]: INFO ql.Context: New scratch dir is hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1
15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Completed getting MetaData in Semantic Analysis
15/12/03 16:18:54 [main]: INFO parse.BaseSemanticAnalyzer: Not invoking CBO because the statement has too few joins
15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30) owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id, i64 data_object_id, varchar(19) object_type, timestamp created, timestamp last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1) temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace, varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2, varchar(32) attribute, i32 op_type, timestamp op_time}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Table Plan for t TS[0]
15/12/03 16:18:54 [main]: DEBUG exec.FunctionRegistry: Looking up GenericUDAF: count
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: RR before GB t{(owner,owner: varchar(30))(object_name,object_name: varchar(30))(subobject_name,subobject_name: varchar(30))(object_id,object_id: bigint)(data_object_id,data_object_id: bigint)(object_type,object_type: varchar(19))(created,created: timestamp)(last_ddl_time,last_ddl_time: timestamp)(timestamp,timestamp: varchar(19))(status,status: varchar(7))(temporary2,temporary2: varchar(1))(generated,generated: varchar(1))(secondary,secondary: varchar(1))(namespace,namespace: bigint)(edition_name,edition_name: varchar(30))(padding1,padding1: varchar(4000))(padding2,padding2: varchar(3500))(attribute,attribute: varchar(32))(op_type,op_type: int)(op_time,op_time: timestamp)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)(row__id,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>)} after GB {((tok_function count 1),_col0: bigint)}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: tree: (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION count 1)))
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: genSelectPlan: input = {((tok_function count 1),_col0: bigint)} starRr = t{(owner,owner: varchar(30))(object_name,object_name: varchar(30))(subobject_name,subobject_name: varchar(30))(object_id,object_id: bigint)(data_object_id,data_object_id: bigint)(object_type,object_type: varchar(19))(created,created: timestamp)(last_ddl_time,last_ddl_time: timestamp)(timestamp,timestamp: varchar(19))(status,status: varchar(7))(temporary2,temporary2: varchar(1))(generated,generated: varchar(1))(secondary,secondary: varchar(1))(namespace,namespace: bigint)(edition_name,edition_name: varchar(30))(padding1,padding1: varchar(4000))(padding2,padding2: varchar(3500))(attribute,attribute: varchar(32))(op_type,op_type: int)(op_time,op_time: timestamp)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)(row__id,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>)}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Select Plan row schema: null{(_c0,_col0: bigint)}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Select Plan for clause: insclause-0
15/12/03 16:18:54 [main]: DEBUG ql.Context: Created staging dir = hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1 for path = hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000
15/12/03 16:18:54 [main]: INFO common.FileUtils: Creating directory if it doesn't exist: hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #11
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #11
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #12
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #12
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #13
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #13
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #14
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #14
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [main]: DEBUG hdfs.DFSClient: /tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1: masked=rwxr-xr-x
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #15
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #15
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: mkdirs took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #16
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #16
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: {-chgrp,-R,supergroup,hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000}
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #17
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #17
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #18
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #18
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #19
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #19
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 1ms
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: Return value is :0
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: {-chmod,-R,700,hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000}
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #20
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #20
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #21
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #21
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: setPermission took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #22
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #22
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 1ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #23
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #23
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: setPermission took 5ms
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #24
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #24
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 0ms
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: Return value is :0
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #25
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #25
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
15/12/03 16:18:54 [main]: DEBUG shims.HadoopShimsSecure: FileStatus{path=hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000; isDirectory=true; modification_time=1449159534181; access_time=0; owner=hduser; group=supergroup; permission=rwx------; isSymlink=false}
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #26
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #26
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 0ms
15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[_col0] columnTypes=[bigint] separator=[[B@51669ea5] nullstring=\N lastColumnTakesRest=false timestampFormats=null
15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[_col0] columnTypes=[bigint] separator=[[B@15a2b981] nullstring=\N lastColumnTakesRest=false timestampFormats=null
15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Set stats collection dir : hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000/.hive-staging_hive_2015-12-03_16-18-53_903_4245610515605755949-1/-ext-10002
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created FileSink Plan for clause: insclause-0dest_path: hdfs://rhes564:9000/tmp/hive/hduser/35f3f0b8-de38-4ece-ab99-d2e94226a0b4/hive_2015-12-03_16-18-53_903_4245610515605755949-1/-mr-10000 row schema: null{(_c0,_col0: bigint)}
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Body Plan for Query Block null
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Created Plan for Query Block null
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: Before logical optimization
TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-SEL[5]-FS[6]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:TS[0] with rs:(owner: varchar(30)|{t}owner,object_name: varchar(30)|{t}object_name,subobject_name: varchar(30)|{t}subobject_name,object_id: bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type: varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time: timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status: varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated: varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace: bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1: varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute: varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time: timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE: bigint|{t}block__offset__inside__file,INPUT__FILE__NAME: string|{t}input__file__name,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator TS[0]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:SEL[1] with rs:(owner: varchar(30)|{t}owner,object_name: varchar(30)|{t}object_name,subobject_name: varchar(30)|{t}subobject_name,object_id: bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type: varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time: timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status: varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated: varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace: bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1: varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute: varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time: timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE: bigint|{t}block__offset__inside__file,INPUT__FILE__NAME: string|{t}input__file__name,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator SEL[1]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory: New column list:(Column[owner] Column[object_name] Column[subobject_name] Column[object_id] Column[data_object_id] Column[object_type] Column[created] Column[last_ddl_time] Column[timestamp] Column[status] Column[temporary2] Column[generated] Column[secondary] Column[namespace] Column[edition_name] Column[padding1] Column[padding2] Column[attribute] Column[op_type] Column[op_time] Column[BLOCK__OFFSET__INSIDE__FILE] Column[INPUT__FILE__NAME] Column[ROW__ID])
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:GBY[2] with rs:(_col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator GBY[2]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:RS[3] with rs:(VALUE._col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator RS[3]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:GBY[4] with rs:(_col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator GBY[4]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:SEL[5] with rs:(_col0: bigint|{null}_c0)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator SEL[5]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory: New column list:(Column[_col0])
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:FS[6] with rs:(_col0: bigint|{})
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator FS[6]
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for FS(6)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for SEL(5)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for GBY(4)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for RS(3)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for GBY(2)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for SEL(1)
15/12/03 16:18:54 [main]: INFO ppd.OpProcFactory: Processing for TS(0)
15/12/03 16:18:54 [main]: DEBUG ppd.PredicatePushDown: After PPD:
TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-SEL[5]-FS[6]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:TS[0] with rs:(owner: varchar(30)|{t}owner,object_name: varchar(30)|{t}object_name,subobject_name: varchar(30)|{t}subobject_name,object_id: bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type: varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time: timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status: varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated: varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace: bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1: varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute: varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time: timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE: bigint|{t}block__offset__inside__file,INPUT__FILE__NAME: string|{t}input__file__name,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator TS[0]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:SEL[1] with rs:(owner: varchar(30)|{t}owner,object_name: varchar(30)|{t}object_name,subobject_name: varchar(30)|{t}subobject_name,object_id: bigint|{t}object_id,data_object_id: bigint|{t}data_object_id,object_type: varchar(19)|{t}object_type,created: timestamp|{t}created,last_ddl_time: timestamp|{t}last_ddl_time,timestamp: varchar(19)|{t}timestamp,status: varchar(7)|{t}status,temporary2: varchar(1)|{t}temporary2,generated: varchar(1)|{t}generated,secondary: varchar(1)|{t}secondary,namespace: bigint|{t}namespace,edition_name: varchar(30)|{t}edition_name,padding1: varchar(4000)|{t}padding1,padding2: varchar(3500)|{t}padding2,attribute: varchar(32)|{t}attribute,op_type: int|{t}op_type,op_time: timestamp|{t}op_time,BLOCK__OFFSET__INSIDE__FILE: bigint|{t}block__offset__inside__file,INPUT__FILE__NAME: string|{t}input__file__name,ROW__ID: struct<transactionid:bigint,bucketid:int,rowid:bigint>|{t}row__id)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator SEL[1]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory: New column list:(Column[owner] Column[object_name] Column[subobject_name] Column[object_id] Column[data_object_id] Column[object_type] Column[created] Column[last_ddl_time] Column[timestamp] Column[status] Column[temporary2] Column[generated] Column[secondary] Column[namespace] Column[edition_name] Column[padding1] Column[padding2] Column[attribute] Column[op_type] Column[op_time] Column[BLOCK__OFFSET__INSIDE__FILE] Column[INPUT__FILE__NAME] Column[ROW__ID])
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:GBY[2] with rs:(_col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator GBY[2]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:RS[3] with rs:(VALUE._col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator RS[3]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:GBY[4] with rs:(_col0: bigint|{}(tok_function count 1))
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator GBY[4]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:SEL[5] with rs:(_col0: bigint|{null}_c0)
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator SEL[5]
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcFactory: New column list:(Column[_col0])
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Getting constants of op:FS[6] with rs:(_col0: bigint|{})
15/12/03 16:18:54 [main]: DEBUG optimizer.ConstantPropagateProcCtx: Offerring constants [] to operator FS[6]
15/12/03 16:18:54 [main]: DEBUG optimizer.ColumnPrunerProcFactory: Reduce Sink Operator 3 key:[]
15/12/03 16:18:54 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 3 oldColExprMap: {VALUE._col0=Column[_col0]}
15/12/03 16:18:54 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 3 newColExprMap: {VALUE._col0=Column[_col0]}
15/12/03 16:18:54 [main]: DEBUG optimizer.IdentityProjectRemover: Identity project remover optimization removed : SEL[5]
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=partition-retrieving from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=partition-retrieving start=1449159534437 end=1449159534440 duration=3 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
15/12/03 16:18:54 [main]: DEBUG exec.TableScanOperator: Setting stats (Num rows: 1767886 Data size: 5984968162 Basic stats: COMPLETE Column stats: COMPLETE) on TS[0]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-TS[0] (t): numRows: 1767886 dataSize: 5984968162 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {}
15/12/03 16:18:54 [main]: DEBUG exec.SelectOperator: Setting stats (Num rows: 1767886 Data size: 5984968162 Basic stats: COMPLETE Column stats: COMPLETE) on SEL[1]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-SEL[1]: numRows: 1767886 dataSize: 5984968162 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {}
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: STATS-GBY[2]: inputSize: 5984968162 maxSplitSize: 256000000 parallelism: 24 containsGroupingSet: false sizeOfGroupingSet: 1
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: STATS-GBY[2] hashAgg: true
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [Case 3] STATS-GBY[2]: cardinality: 24
15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting stats (Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE) on GBY[2]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-GBY[2]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {_col0= colName: _col0 colType: bigint countDistincts: 24 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false}
15/12/03 16:18:54 [main]: DEBUG exec.ReduceSinkOperator: Setting stats (Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE) on RS[3]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-RS[3]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {VALUE._col0= colName: VALUE._col0 colType: bigint countDistincts: 24 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false}
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: STATS-GBY[4]: inputSize: 1 maxSplitSize: 256000000 parallelism: 1 containsGroupingSet: false sizeOfGroupingSet: 1
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: STATS-GBY[4] hashAgg: false
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [Case 9] STATS-GBY[4]: cardinality: 1
15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting stats (Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE) on GBY[4]
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-GBY[4]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {_col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false}
15/12/03 16:18:54 [main]: DEBUG annotation.StatsRulesProcFactory: [0] STATS-FS[6]: numRows: 1 dataSize: 8 basicStatsState: COMPLETE colStatsState: COMPLETE colStats: {_col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false}
15/12/03 16:18:54 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #27
15/12/03 16:18:54 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #27
15/12/03 16:18:54 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getListing took 1ms
15/12/03 16:18:54 [main]: DEBUG exec.TableScanOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@34dccb3d) on TS[0]
15/12/03 16:18:54 [main]: DEBUG exec.SelectOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@ba32ce8) on SEL[1]
15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@5d93950) on GBY[2]
15/12/03 16:18:54 [main]: DEBUG exec.ReduceSinkOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@7841b62f) on RS[3]
15/12/03 16:18:54 [main]: DEBUG exec.GroupByOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@22200557) on GBY[4]
15/12/03 16:18:54 [main]: DEBUG exec.FileSinkOperator: Setting traits (org.apache.hadoop.hive.ql.plan.OpTraits@22200557) on FS[6]
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: After logical optimization
TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-FS[6]
15/12/03 16:18:54 [main]: DEBUG security.Groups: Returning cached groups for 'hduser'
15/12/03 16:18:54 [main]: DEBUG security.Groups: Returning cached groups for 'hduser'
15/12/03 16:18:54 [main]: DEBUG session.SessionState: Session is using authorization class class org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=SparkOptimizeOperatorTree from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: INFO spark.SetSparkReducerParallelism: Number of reducers determined to be: 1
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=SparkOptimizeOperatorTree start=1449159534470 end=1449159534477 duration=7 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=SparkGenerateTaskTree from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Root operator: TS[0]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Leaf operator: RS[3]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkUtils: Adding map work (Map 1) for TS[0]
15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30) owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id, i64 data_object_id, varchar(19) object_type, timestamp created, timestamp last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1) temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace, varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2, varchar(32) attribute, i32 op_type, timestamp op_time}
15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30) owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id, i64 data_object_id, varchar(19) object_type, timestamp created, timestamp last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1) temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace, varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2, varchar(32) attribute, i32 op_type, timestamp op_time}
15/12/03 16:18:54 [main]: DEBUG hive.log: DDL: struct t { varchar(30) owner, varchar(30) object_name, varchar(30) subobject_name, i64 object_id, i64 data_object_id, varchar(19) object_type, timestamp created, timestamp last_ddl_time, varchar(19) timestamp, varchar(7) status, varchar(1) temporary2, varchar(1) generated, varchar(1) secondary, i64 namespace, varchar(30) edition_name, varchar(4000) padding1, varchar(3500) padding2, varchar(32) attribute, i32 op_type, timestamp op_time}
15/12/03 16:18:54 [main]: DEBUG optimizer.GenMapRedUtils: Adding hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t of tablet
15/12/03 16:18:54 [main]: DEBUG optimizer.GenMapRedUtils: Information added for path hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: First pass. Leaf operator: RS[3]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Root operator: GBY[4]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Leaf operator: FS[6]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkUtils: Adding reduce work (Reducer 2) for GBY[4]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkUtils: Setting up reduce sink: RS[3] with following reduce work: Reducer 2
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: Removing RS[3] as parent from GBY[4]
15/12/03 16:18:54 [main]: DEBUG spark.GenSparkWork: First pass. Leaf operator: FS[6]
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=SparkGenerateTaskTree start=1449159534477 end=1449159534495 duration=18 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=SparkOptimizeTaskTree from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: DEBUG parse.TaskCompiler: Skipping runtime skew join optimization
15/12/03 16:18:54 [main]: DEBUG physical.NullScanTaskDispatcher: Looking at: Map 1
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
15/12/03 16:18:54 [main]: DEBUG physical.NullScanTaskDispatcher: Looking at: Map 1
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
15/12/03 16:18:54 [main]: DEBUG physical.NullScanTaskDispatcher: Looking at: Map 1
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
15/12/03 16:18:54 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
15/12/03 16:18:54 [main]: DEBUG parse.TaskCompiler: Skipping vectorization
15/12/03 16:18:54 [main]: DEBUG parse.TaskCompiler: Skipping stage id rearranger
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=SparkOptimizeTaskTree start=1449159534495 end=1449159534510 duration=15 from=org.apache.hadoop.hive.ql.parse.spark.SparkCompiler>
15/12/03 16:18:54 [main]: INFO parse.CalcitePlanner: Completed plan generation
15/12/03 16:18:54 [main]: INFO ql.Driver: Semantic Analysis Completed
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: validation start
15/12/03 16:18:54 [main]: DEBUG parse.CalcitePlanner: not validating writeEntity, because entity is neither table nor partition
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=semanticAnalyze start=1449159533916 end=1449159534511 duration=595 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[_col0] columnTypes=[bigint] separator=[[B@710bed9c] nullstring=\N lastColumnTakesRest=false timestampFormats=null
15/12/03 16:18:54 [main]: INFO exec.ListSinkOperator: Initializing operator OP[7]
15/12/03 16:18:54 [main]: DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.DelimitedJSONSerDe initialized with: columnNames=[] columnTypes=[] separator=[[B@7ead8de4] nullstring=NULL lastColumnTakesRest=false timestampFormats=null
15/12/03 16:18:54 [main]: INFO exec.ListSinkOperator: Initialization Done 7 OP
15/12/03 16:18:54 [main]: INFO exec.ListSinkOperator: Operator 7 OP initialized
15/12/03 16:18:54 [main]: INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=compile start=1449159533903 end=1449159534532 duration=629 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: INFO metadata.Hive: Dumping metastore api call timing information for : compilation phase
15/12/03 16:18:54 [main]: DEBUG metadata.Hive: Total time spent in each metastore function (ms): {isCompatibleWith_(HiveConf, )=1, getTable_(String, String, )=37}
15/12/03 16:18:54 [main]: INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: INFO ql.Driver: Starting command(queryId=hduser_20151203161853_0e621371-6907-430e-856a-998d32d0c314): select count(1) from t
Query ID = hduser_20151203161853_0e621371-6907-430e-856a-998d32d0c314
15/12/03 16:18:54 [main]: INFO ql.Driver: Query ID = hduser_20151203161853_0e621371-6907-430e-856a-998d32d0c314
Total jobs = 1
15/12/03 16:18:54 [main]: INFO ql.Driver: Total jobs = 1
15/12/03 16:18:54 [main]: INFO log.PerfLogger: </PERFLOG method=TimeToSubmit start=1449159533903 end=1449159534532 duration=629 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:18:54 [main]: INFO log.PerfLogger: <PERFLOG method=task.SPARK.Stage-1 from=org.apache.hadoop.hive.ql.Driver>
Launching Job 1 out of 1
15/12/03 16:18:54 [main]: INFO ql.Driver: Launching Job 1 out of 1
15/12/03 16:18:54 [main]: INFO ql.Driver: Starting task [Stage-1:MAPRED] in serial mode
In order to change the average load for a reducer (in bytes):
15/12/03 16:18:54 [main]: INFO spark.SparkTask: In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
15/12/03 16:18:54 [main]: INFO spark.SparkTask: set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
15/12/03 16:18:54 [main]: INFO spark.SparkTask: In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
15/12/03 16:18:54 [main]: INFO spark.SparkTask: set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
15/12/03 16:18:54 [main]: INFO spark.SparkTask: In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
15/12/03 16:18:54 [main]: INFO spark.SparkTask: set mapreduce.job.reduces=<number>
15/12/03 16:18:54 [main]: INFO session.SparkSessionManagerImpl: Setting up the session manager.
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.connect.timeout -> 1000).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.serializer -> org.apache.spark.serializer.KryoSerializer).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.eventLog.enabled -> true).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.eventLog.dir -> /usr/lib/spark/logs).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.rpc.threads -> 8).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.secret.bits -> 256).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.home -> /usr/lib/spark).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.rpc.max.size -> 52428800).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.master -> spark://rhes564:7077).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.executor.memory -> 512m).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.server.connect.timeout -> 90000).
15/12/03 16:18:54 [main]: DEBUG logging.InternalLoggerFactory: Using SLF4J as the default logging framework
15/12/03 16:18:54 [main]: DEBUG channel.MultithreadEventLoopGroup: -Dio.netty.eventLoopThreads: 24
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0: java.nio.Buffer.address: available
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0: sun.misc.Unsafe.theUnsafe: available
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0: sun.misc.Unsafe.copyMemory: available
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent0: java.nio.Bits.unaligned: true
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: UID: 1009
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: Java version: 7
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.noUnsafe: false
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: sun.misc.Unsafe: available
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.noJavassist: false
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: Javassist: unavailable
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.bitMode: 64 (sun.arch.data.model)
15/12/03 16:18:54 [main]: DEBUG internal.PlatformDependent: -Dio.netty.noPreferDirect: false
15/12/03 16:18:54 [main]: DEBUG nio.NioEventLoop: -Dio.netty.noKeySetOptimization: false
15/12/03 16:18:54 [main]: DEBUG nio.NioEventLoop: -Dio.netty.selectorAutoRebuildThreshold: 512
15/12/03 16:18:54 [main]: DEBUG internal.ThreadLocalRandom: -Dio.netty.initialSeedUniquifier: 0x0b4453d9b0adc2b0 (took 0 ms)
15/12/03 16:18:54 [main]: DEBUG buffer.ByteBufUtil: -Dio.netty.allocator.type: unpooled
15/12/03 16:18:54 [main]: DEBUG buffer.ByteBufUtil: -Dio.netty.threadLocalDirectBufferSize: 65536
15/12/03 16:18:54 [main]: DEBUG util.NetUtil: Loopback interface: lo (lo, 127.0.0.1)
15/12/03 16:18:54 [main]: DEBUG util.NetUtil: /proc/sys/net/core/somaxconn: 128
15/12/03 16:18:54 [main]: WARN rpc.RpcConfiguration: Your hostname, rhes564, resolves to a loopback address, but we couldn't find any external IP address!
15/12/03 16:18:54 [main]: WARN rpc.RpcConfiguration: Set hive.spark.client.server.address if you need to bind to another address.
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.connect.timeout -> 1000).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.serializer -> org.apache.spark.serializer.KryoSerializer).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.eventLog.enabled -> true).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.eventLog.dir -> /usr/lib/spark/logs).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.rpc.threads -> 8).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.secret.bits -> 256).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.home -> /usr/lib/spark).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.rpc.max.size -> 52428800).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.master -> spark://rhes564:7077).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load spark property from hive configuration (spark.executor.memory -> 512m).
15/12/03 16:18:54 [main]: INFO spark.HiveSparkClientFactory: load RPC property from hive configuration (hive.spark.client.server.connect.timeout -> 90000).
15/12/03 16:18:54 [main]: INFO client.SparkClientImpl: Running client driver with argv: /usr/lib/spark/bin/spark-submit --properties-file /tmp/spark-submit.4668566717444853712.properties --class org.apache.hive.spark.client.RemoteDriver /usr/lib/hive/lib/hive-exec-1.2.1.jar --remote-host rhes564 --remote-port 36561 --conf hive.spark.client.connect.timeout=1000 --conf hive.spark.client.server.connect.timeout=90000 --conf hive.spark.client.channel.log.level=null --conf hive.spark.client.rpc.max.size=52428800 --conf hive.spark.client.rpc.threads=8 --conf hive.spark.client.secret.bits=256
15/12/03 16:18:55 [stderr-redir-1]: INFO client.SparkClientImpl: Spark assembly has been built with Hive, including Datanucleus jars on classpath
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.connect.timeout=1000
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.rpc.threads=8
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.rpc.max.size=52428800
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.secret.bits=256
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Warning: Ignoring non-spark config property: hive.spark.client.server.connect.timeout=90000
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/03 16:18:56 INFO client.RemoteDriver: Connecting to: rhes564:36561
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_CLIENT_CONNECT_TIMEOUT
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:46)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:146)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at java.lang.reflect.Method.invoke(Method.java:606)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
15/12/03 16:18:56 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/12/03 16:18:57 [Driver]: WARN client.SparkClientImpl: Child process exited with code 1.
15/12/03 16:19:04 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: closed
15/12/03 16:19:04 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: stopped, remaining connections 0
15/12/03 16:20:24 [main]: WARN client.SparkClientImpl: Error while waiting for client to connect.
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:724)
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
15/12/03 16:20:24 [main]: ERROR spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:109)
at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
... 21 more
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
... 25 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:724)
15/12/03 16:20:24 [main]: ERROR spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:109)
at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:90)
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:65)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
... 21 more
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:99)
... 25 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:123)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:724)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
15/12/03 16:20:24 [main]: ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
15/12/03 16:20:24 [main]: DEBUG ql.Driver: Shutting down query select count(1) from t
15/12/03 16:20:24 [main]: INFO metadata.Hive: Dumping metastore api call timing information for : execution phase
15/12/03 16:20:24 [main]: DEBUG metadata.Hive: Total time spent in each metastore function (ms): {isCompatibleWith_(HiveConf, )=0}
15/12/03 16:20:24 [main]: INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1449159534532 end=1449159624872 duration=90340 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:20:24 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:20:24 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1449159624872 end=1449159624872 duration=0 from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:20:24 [main]: INFO exec.ListSinkOperator: 7 finished. closing...
15/12/03 16:20:24 [main]: INFO exec.ListSinkOperator: 7 Close done
15/12/03 16:20:24 [main]: DEBUG ql.Driver: Shutting down query select count(1) from t
15/12/03 16:20:24 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
15/12/03 16:20:24 [main]: DEBUG ipc.Client: Connecting to rhes564/50.140.197.217:9000
15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: starting, having connections 1
15/12/03 16:20:24 [IPC Parameter Sending Thread #1]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #28
15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #28
15/12/03 16:20:24 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 7ms
15/12/03 16:20:24 [IPC Parameter Sending Thread #1]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #29
15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #29
15/12/03 16:20:24 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 1ms
15/12/03 16:20:24 [IPC Parameter Sending Thread #1]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser sending #30
15/12/03 16:20:24 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser got value #30
15/12/03 16:20:24 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 2ms
15/12/03 16:20:24 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
15/12/03 16:20:24 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1449159624886 end=1449159624886 duration=0 from=org.apache.hadoop.hive.ql.Driver>
hive>
>
>
>
>
> 15/12/03 16:20:34 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: closed
15/12/03 16:20:34 [IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser]: DEBUG ipc.Client: IPC Client (533248865) connection to rhes564/50.140.197.217:9000 from hduser: stopped, remaining connections 0
Mich Talebzadeh
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Xuefu Zhang [mailto:xzhang@cloudera.com]
Sent: 03 December 2015 14:02
To: user@hive.apache.org
Cc: user@spark.apache.org
Subject: Re: Building spark 1.3 from source code to work with Hive 1.2.1
Mich,
To start your Spark standalone cluster, you can just download the tarball from Spark repo site. In other words, you don't need to start your cluster using your build.
You only need to spark-assembly.jar to Hive's /lib directory and that's it.
I guess you have been confused by this, which I tried to explain previously.
Thanks,
Xuefu
On Thu, Dec 3, 2015 at 2:28 AM, Mich Talebzadeh <mich@peridale.co.uk <ma...@peridale.co.uk> > wrote:
Hi,
I have seen mails that state that the user has managed to build spark 1.3 to work with Hive. I tried Spark 1.5.2 but no luck
I downloaded spark source 1.3 source code spark-1.3.0.tar and built it as follows
./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"
This successfully completed and created the tarred zip file. I then created spark 1.3 tree from this zipped file. $SPARK_HOME is /usr/lib/spark
Other steps that I performed:
1. In $HIVE_HOME/lib , I copied spark-assembly-1.3.0-hadoop2.4.0.jar to this directory
2. In $SPARK_HOME/conf I created a syblink to /usr/lib/hive/conf/hive-site.xml
Then I tried to start spark master node
/usr/lib/spark/sbin/start-master.sh
I get the following error:
cat /usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out
Spark Command: /usr/java/latest/bin/java -cp :/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/home/hduser/hadoop-2.6.0/etc/hadoop -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port 8080
========================================
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
at java.lang.Class.getMethod0(Class.java:2764)
at java.lang.Class.getMethod(Class.java:1653)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I also notice that in /usr/lib/spark/lib, I only have the following jar files
-rw-r--r-- 1 hduser hadoop 98795479 Dec 3 09:03 spark-examples-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 98187168 Dec 3 09:03 spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 4136760 Dec 3 09:03 spark-1.3.0-yarn-shuffle.jar
Wheras in pre-build downloaded one --> /usr/lib/spark-1.3.0-bin-hadoop2.4, there are additional JAR files
-rw-rw-r-- 1 hduser hadoop 1890075 Mar 6 2015 datanucleus-core-3.2.10.jar
-rw-rw-r-- 1 hduser hadoop 112446389 Mar 6 2015 spark-examples-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 159319006 Mar 6 2015 spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 4136744 Mar 6 2015 spark-1.3.0-yarn-shuffle.jar
-rw-rw-r-- 1 hduser hadoop 1809447 Mar 6 2015 datanucleus-rdbms-3.2.9.jar
-rw-rw-r-- 1 hduser hadoop 339666 Mar 6 2015 datanucleus-api-jdo-3.2.6.jar
Any ideas what is is missing? I am sure someone has sorted this one out before.
Thanks,
Mich
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
Re: Building spark 1.3 from source code to work with Hive 1.2.1
Posted by Xuefu Zhang <xz...@cloudera.com>.
Mich,
To start your Spark standalone cluster, you can just download the tarball
from Spark repo site. In other words, you don't need to start your cluster
using your build.
You only need to spark-assembly.jar to Hive's /lib directory and that's it.
I guess you have been confused by this, which I tried to explain previously.
Thanks,
Xuefu
On Thu, Dec 3, 2015 at 2:28 AM, Mich Talebzadeh <mi...@peridale.co.uk> wrote:
> Hi,
>
>
>
> I have seen mails that state that the user has managed to build spark 1.3
> to work with Hive. I tried Spark 1.5.2 but no luck
>
>
>
> I downloaded spark source 1.3 source code spark-1.3.0.tar and built it as
> follows
>
>
>
> ./make-distribution.sh --name "hadoop2-without-hive" --tgz
> "-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"
>
>
>
> This successfully completed and created the tarred zip file. I then
> created spark 1.3 tree from this zipped file. $SPARK_HOME is /
> usr/lib/spark
>
>
>
> Other steps that I performed:
>
>
>
> 1. In $HIVE_HOME/lib , I copied spark-assembly-1.3.0-hadoop2.4.0.jar to
> this directory
>
> 2. In $SPARK_HOME/conf I created a syblink to
> /usr/lib/hive/conf/hive-site.xml
>
>
>
> Then I tried to start spark master node
>
>
>
> /usr/lib/spark/sbin/start-master.sh
>
>
>
> I get the following error:
>
>
>
>
>
> cat
> /usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out
>
> Spark Command: /usr/java/latest/bin/java -cp
> :/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/home/hduser/hadoop-2.6.0/etc/hadoop
> -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m
> org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port
> 8080
>
> ========================================
>
>
>
> Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
>
> at java.lang.Class.getDeclaredMethods0(Native Method)
>
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
>
> at java.lang.Class.getMethod0(Class.java:2764)
>
> at java.lang.Class.getMethod(Class.java:1653)
>
> at
> sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
>
> at
> sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
>
> Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>
>
> I also notice that in /usr/lib/spark/lib, I only have the following jar
> files
>
>
>
> -rw-r--r-- 1 hduser hadoop 98795479 Dec 3 09:03
> spark-examples-1.3.0-hadoop2.4.0.jar
>
> -rw-r--r-- 1 hduser hadoop 98187168 Dec 3 09:03
> spark-assembly-1.3.0-hadoop2.4.0.jar
>
> -rw-r--r-- 1 hduser hadoop 4136760 Dec 3 09:03
> spark-1.3.0-yarn-shuffle.jar
>
>
>
> Wheras in pre-build downloaded one à /usr/lib/spark-1.3.0-bin-hadoop2.4, there
> are additional JAR files
>
>
>
> -rw-rw-r-- 1 hduser hadoop 1890075 Mar 6 2015
> datanucleus-core-3.2.10.jar
>
> -rw-rw-r-- 1 hduser hadoop 112446389 Mar 6 2015
> spark-examples-1.3.0-hadoop2.4.0.jar
>
> -rw-rw-r-- 1 hduser hadoop 159319006 Mar 6 2015
> spark-assembly-1.3.0-hadoop2.4.0.jar
>
> -rw-rw-r-- 1 hduser hadoop 4136744 Mar 6 2015
> spark-1.3.0-yarn-shuffle.jar
>
> -rw-rw-r-- 1 hduser hadoop 1809447 Mar 6 2015
> datanucleus-rdbms-3.2.9.jar
>
> -rw-rw-r-- 1 hduser hadoop 339666 Mar 6 2015
> datanucleus-api-jdo-3.2.6.jar
>
>
>
> Any ideas what is is missing? I am sure someone has sorted this one out
> before.
>
>
>
>
>
> Thanks,
>
>
>
> Mich
>
>
>
>
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Ltd, its subsidiaries nor their employees
> accept any responsibility.
>
>
>
Re: Building spark 1.3 from source code to work with Hive 1.2.1
Posted by zhangjp <59...@qq.com>.
I have encountered the same issues. before I changed the spark version i setted up environment as follows.
spark 1.5.2
hadoop 2.6.2
hive 1.2.1
but no luck it's not work well, even through i run essembly hive in spark with jdbc mode there is also some properblems.
then I changed the spark version 1.3.1 and rebuild, I just run a example ,there is a issuse seems the pb version conflict, will rebuild and try again.....
Exception in thread "main" java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$PriorityProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
------------------ Original ------------------
From: "Mich Talebzadeh";<mi...@peridale.co.uk>;
Date: Thu, Dec 3, 2015 06:28 PM
To: "user"<us...@hive.apache.org>; "user"<us...@spark.apache.org>;
Subject: Building spark 1.3 from source code to work with Hive 1.2.1
Hi,
I have seen mails that state that the user has managed to build spark 1.3 to work with Hive. I tried Spark 1.5.2 but no luck
I downloaded spark source 1.3 source code spark-1.3.0.tar and built it as follows
./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"
This successfully completed and created the tarred zip file. I then created spark 1.3 tree from this zipped file. $SPARK_HOME is /usr/lib/spark
Other steps that I performed:
1. In $HIVE_HOME/lib , I copied spark-assembly-1.3.0-hadoop2.4.0.jar to this directory
2. In $SPARK_HOME/conf I created a syblink to /usr/lib/hive/conf/hive-site.xml
Then I tried to start spark master node
/usr/lib/spark/sbin/start-master.sh
I get the following error:
cat /usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out
Spark Command: /usr/java/latest/bin/java -cp :/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/home/hduser/hadoop-2.6.0/etc/hadoop -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port 8080
========================================
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
at java.lang.Class.getMethod0(Class.java:2764)
at java.lang.Class.getMethod(Class.java:1653)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I also notice that in /usr/lib/spark/lib, I only have the following jar files
-rw-r--r-- 1 hduser hadoop 98795479 Dec 3 09:03 spark-examples-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 98187168 Dec 3 09:03 spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 4136760 Dec 3 09:03 spark-1.3.0-yarn-shuffle.jar
Wheras in pre-build downloaded one à /usr/lib/spark-1.3.0-bin-hadoop2.4, there are additional JAR files
-rw-rw-r-- 1 hduser hadoop 1890075 Mar 6 2015 datanucleus-core-3.2.10.jar
-rw-rw-r-- 1 hduser hadoop 112446389 Mar 6 2015 spark-examples-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 159319006 Mar 6 2015 spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 4136744 Mar 6 2015 spark-1.3.0-yarn-shuffle.jar
-rw-rw-r-- 1 hduser hadoop 1809447 Mar 6 2015 datanucleus-rdbms-3.2.9.jar
-rw-rw-r-- 1 hduser hadoop 339666 Mar 6 2015 datanucleus-api-jdo-3.2.6.jar
Any ideas what is is missing? I am sure someone has sorted this one out before.
Thanks,
Mich
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.