You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kevin Paul <ke...@gmail.com> on 2014/10/03 10:43:41 UTC
SparkSQL on Hive error
Hi all, I tried to launch my application with spark-submit, the command I
use is:
bin/spark-submit --class ${MY_CLASS} --jars ${MY_JARS} --master local
myApplicationJar.jar
I've buillt spark with SPARK_HIVE=true, and was able to start HiveContext,
and was able to run command like,
hiveContext.sql("select * from myTable")
or hiveContext.sql("select count(*) from myTable")
myTable is a table on my hive database. However, when I run the command:
hiveContext.sql("show tables"), I got the following error:
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.Driver.validateConfVariables(Driver.java:1057)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:948)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:298)
at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:272)
at
org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
at
org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
at
org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38)
at
org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360)
at
org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360)
Thanks,
Kelvin Paul
Re: SparkSQL on Hive error
Posted by Kevin Paul <ke...@gmail.com>.
Thanks Michael, your patch works for me :)
Regards,
Kelvin Paul
On Fri, Oct 3, 2014 at 3:52 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> Are you running master? There was briefly a regression here that is
> hopefully fixed by spark#2635 <https://github.com/apache/spark/pull/2635>.
>
> On Fri, Oct 3, 2014 at 1:43 AM, Kevin Paul <ke...@gmail.com>
> wrote:
>
>> Hi all, I tried to launch my application with spark-submit, the command I
>> use is:
>>
>> bin/spark-submit --class ${MY_CLASS} --jars ${MY_JARS} --master local
>> myApplicationJar.jar
>>
>> I've buillt spark with SPARK_HIVE=true, and was able to start
>> HiveContext, and was able to run command like,
>> hiveContext.sql("select * from myTable")
>> or hiveContext.sql("select count(*) from myTable")
>> myTable is a table on my hive database. However, when I run the command:
>> hiveContext.sql("show tables"), I got the following error:
>>
>> java.lang.NullPointerException
>>
>> at
>> org.apache.hadoop.hive.ql.Driver.validateConfVariables(Driver.java:1057)
>>
>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:948)
>>
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
>>
>> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:298)
>>
>> at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:272)
>>
>> at
>> org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
>>
>> at
>> org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
>>
>> at
>> org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38)
>>
>> at
>> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360)
>>
>> at
>> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360)
>>
>>
>> Thanks,
>> Kelvin Paul
>>
>
>
How to save Spark log into file
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
How can the spark log be saved into file instead of showing them on console?
Below is my conf/log4j.properties
conf/log4j.properties
###
# Root logger option
log4j.rootLogger=INFO, file
# Direct log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
#Redirect to Tomcat logs folder
log4j.appender.file.File=/hadoop_logs/spark/spark_logging.log
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
###
I tried to stop and start spark again, it still shows INFO WARN log on console.
Any ideas?
Regards
Arthur
scala> hiveContext.hql("show tables")
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
2014-10-03 19:35:01,554 INFO [main] parse.ParseDriver (ParseDriver.java:parse(179)) - Parsing command: show tables
2014-10-03 19:35:01,715 INFO [main] parse.ParseDriver (ParseDriver.java:parse(197)) - Parse Completed
2014-10-03 19:35:01,845 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-10-03 19:35:01,847 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=Driver.run>
2014-10-03 19:35:01,847 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=TimeToSubmit>
2014-10-03 19:35:01,847 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=compile>
2014-10-03 19:35:01,863 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=parse>
2014-10-03 19:35:01,863 INFO [main] parse.ParseDriver (ParseDriver.java:parse(179)) - Parsing command: show tables
2014-10-03 19:35:01,863 INFO [main] parse.ParseDriver (ParseDriver.java:parse(197)) - Parse Completed
2014-10-03 19:35:01,863 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=parse start=1412336101863 end=1412336101863 duration=0>
2014-10-03 19:35:01,863 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=semanticAnalyze>
2014-10-03 19:35:01,941 INFO [main] ql.Driver (Driver.java:compile(450)) - Semantic Analysis Completed
2014-10-03 19:35:01,941 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=semanticAnalyze start=1412336101863 end=1412336101941 duration=78>
2014-10-03 19:35:01,979 INFO [main] exec.ListSinkOperator (Operator.java:initialize(338)) - Initializing Self 0 OP
2014-10-03 19:35:01,980 INFO [main] exec.ListSinkOperator (Operator.java:initializeChildren(403)) - Operator 0 OP initialized
2014-10-03 19:35:01,980 INFO [main] exec.ListSinkOperator (Operator.java:initialize(378)) - Initialization Done 0 OP
2014-10-03 19:35:01,985 INFO [main] ql.Driver (Driver.java:getSchema(264)) - Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from deserializer)], properties:null)
2014-10-03 19:35:01,985 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=compile start=1412336101847 end=1412336101985 duration=138>
2014-10-03 19:35:01,985 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=Driver.execute>
2014-10-03 19:35:01,985 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.job.name is deprecated. Instead, use mapreduce.job.name
2014-10-03 19:35:01,986 INFO [main] ql.Driver (Driver.java:execute(1117)) - Starting command: show tables
2014-10-03 19:35:01,994 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=TimeToSubmit start=1412336101847 end=1412336101994 duration=147>
2014-10-03 19:35:01,994 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=runTasks>
2014-10-03 19:35:01,994 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=task.DDL.Stage-0>
2014-10-03 19:35:02,019 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(411)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2014-10-03 19:35:02,034 INFO [main] metastore.ObjectStore (ObjectStore.java:initialize(232)) - ObjectStore, initialize called
2014-10-03 19:35:02,084 WARN [main] DataNucleus.General (Log4JLogger.java:warn(96)) - Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file://hadoop/spark/lib_managed/jars/datanucleus-api-jdo-3.2.1.jar" is already registered, and you are trying to register an identical plugin located at URL "file://hadoop/spark-1.1.0_patched/lib_managed/jars/datanucleus-api-jdo-3.2.1.jar."
2014-10-03 19:35:02,108 WARN [main] DataNucleus.General (Log4JLogger.java:warn(96)) - Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file://hadoop/spark/lib_managed/jars/datanucleus-core-3.2.2.jar" is already registered, and you are trying to register an identical plugin located at URL "file://hadoop/spark-1.1.0_patched/lib_managed/jars/datanucleus-core-3.2.2.jar."
2014-10-03 19:35:02,137 WARN [main] DataNucleus.General (Log4JLogger.java:warn(96)) - Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file://hadoop/spark-1.1.0_patched/lib_managed/jars/datanucleus-rdbms-3.2.1.jar" is already registered, and you are trying to register an identical plugin located at URL "file://hadoop/spark/lib_managed/jars/datanucleus-rdbms-3.2.1.jar."
2014-10-03 19:35:02,174 INFO [main] DataNucleus.Persistence (Log4JLogger.java:info(77)) - Property datanucleus.cache.level2 unknown - will be ignored
2014-10-03 19:35:02,300 WARN [main] bonecp.BoneCPConfig (BoneCPConfig.java:sanitize(1537)) - Max Connections < 1. Setting to 20
2014-10-03 19:35:02,769 INFO [main] metastore.ObjectStore (ObjectStore.java:getPMF(280)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2014-10-03 19:35:02,791 INFO [main] metastore.ObjectStore (ObjectStore.java:setConf(215)) - Initialized ObjectStore
2014-10-03 19:35:03,268 WARN [main] bonecp.BoneCPConfig (BoneCPConfig.java:sanitize(1537)) - Max Connections < 1. Setting to 20
2014-10-03 19:35:03,400 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 0: get_database: default
2014-10-03 19:35:03,400 INFO [main] HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=edhuser ip=unknown-ip-addr cmd=get_database: default
2014-10-03 19:35:03,405 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 0: get_tables: db=default pat=.*
2014-10-03 19:35:03,405 INFO [main] HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=edhuser ip=unknown-ip-addr cmd=get_tables: db=default pat=.*
2014-10-03 19:35:03,408 INFO [main] DataNucleus.Datastore (Log4JLogger.java:info(77)) - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
2014-10-03 19:35:03,409 INFO [main] DataNucleus.Datastore (Log4JLogger.java:info(77)) - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
2014-10-03 19:35:03,493 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=task.DDL.Stage-0 start=1412336101994 end=1412336103493 duration=1499>
2014-10-03 19:35:03,493 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=runTasks start=1412336101994 end=1412336103493 duration=1499>
2014-10-03 19:35:03,493 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=Driver.execute start=1412336101985 end=1412336103493 duration=1508>
2014-10-03 19:35:03,496 INFO [main] ql.Driver (SessionState.java:printInfo(410)) - OK
2014-10-03 19:35:03,496 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=releaseLocks>
2014-10-03 19:35:03,496 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=releaseLocks start=1412336103496 end=1412336103496 duration=0>
2014-10-03 19:35:03,496 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=Driver.run start=1412336101847 end=1412336103496 duration=1649>
2014-10-03 19:35:03,497 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-10-03 19:35:03,497 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2014-10-03 19:35:03,508 INFO [main] mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2014-10-03 19:35:03,517 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=releaseLocks>
2014-10-03 19:35:03,518 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=releaseLocks start=1412336103517 end=1412336103518 duration=1>
Re: SparkSQL on Hive error
Posted by Cheng Lian <li...@gmail.com>.
Also make sure to call |hiveContext.sql| within the same thread where
|hiveContext| is created, because Hive uses thread-local variable to
initialize the |Driver.conf|.
On 10/3/14 4:52 PM, Michael Armbrust wrote:
> Are you running master? There was briefly a regression here that is
> hopefully fixed by spark#2635 <https://github.com/apache/spark/pull/2635>.
>
> On Fri, Oct 3, 2014 at 1:43 AM, Kevin Paul <kevinpaulapple@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi all, I tried to launch my application with spark-submit, the
> command I use is:
>
> bin/spark-submit --class ${MY_CLASS} --jars ${MY_JARS} --master
> local myApplicationJar.jar
>
> I've buillt spark with SPARK_HIVE=true, and was able to start
> HiveContext, and was able to run command like,
> hiveContext.sql("select * from myTable")
> or hiveContext.sql("select count(*) from myTable")
> myTable is a table on my hive database. However, when I run the
> command: hiveContext.sql("show tables"), I got the following error:
>
> java.lang.NullPointerException
>
> at
> org.apache.hadoop.hive.ql.Driver.validateConfVariables(Driver.java:1057)
>
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:948)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
>
> at
> org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:298)
>
> at
> org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:272)
>
> at
> org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
>
> at
> org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
>
> at
> org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38)
>
> at
> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360)
>
> at
> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360)
>
>
> Thanks,
> Kelvin Paul
>
>
Re: SparkSQL on Hive error
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
hi,
I have just tested the same command, it works here, can you please provide your create table command?
regards
Arthur
scala> hiveContext.hql("show tables")
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
2014-10-03 17:14:33,575 INFO [main] parse.ParseDriver (ParseDriver.java:parse(179)) - Parsing command: show tables
2014-10-03 17:14:33,575 INFO [main] parse.ParseDriver (ParseDriver.java:parse(197)) - Parse Completed
2014-10-03 17:14:33,600 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-10-03 17:14:33,600 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=Driver.run>
2014-10-03 17:14:33,600 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=TimeToSubmit>
2014-10-03 17:14:33,600 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=compile>
2014-10-03 17:14:33,600 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=parse>
2014-10-03 17:14:33,600 INFO [main] parse.ParseDriver (ParseDriver.java:parse(179)) - Parsing command: show tables
2014-10-03 17:14:33,600 INFO [main] parse.ParseDriver (ParseDriver.java:parse(197)) - Parse Completed
2014-10-03 17:14:33,601 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=parse start=1412327673600 end=1412327673601 duration=1>
2014-10-03 17:14:33,601 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=semanticAnalyze>
2014-10-03 17:14:33,603 INFO [main] ql.Driver (Driver.java:compile(450)) - Semantic Analysis Completed
2014-10-03 17:14:33,603 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=semanticAnalyze start=1412327673601 end=1412327673603 duration=2>
2014-10-03 17:14:33,604 INFO [main] exec.ListSinkOperator (Operator.java:initialize(338)) - Initializing Self 0 OP
2014-10-03 17:14:33,604 INFO [main] exec.ListSinkOperator (Operator.java:initializeChildren(403)) - Operator 0 OP initialized
2014-10-03 17:14:33,604 INFO [main] exec.ListSinkOperator (Operator.java:initialize(378)) - Initialization Done 0 OP
2014-10-03 17:14:33,604 INFO [main] ql.Driver (Driver.java:getSchema(264)) - Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from deserializer)], properties:null)
2014-10-03 17:14:33,605 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=compile start=1412327673600 end=1412327673605 duration=5>
2014-10-03 17:14:33,605 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=Driver.execute>
2014-10-03 17:14:33,605 INFO [main] ql.Driver (Driver.java:execute(1117)) - Starting command: show tables
2014-10-03 17:14:33,605 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=TimeToSubmit start=1412327673600 end=1412327673605 duration=5>
2014-10-03 17:14:33,605 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=runTasks>
2014-10-03 17:14:33,605 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=task.DDL.Stage-0>
2014-10-03 17:14:33,606 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 0: get_database: default
2014-10-03 17:14:33,606 INFO [main] HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=edhuser ip=unknown-ip-addr cmd=get_database: default
2014-10-03 17:14:33,609 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 0: get_tables: db=default pat=.*
2014-10-03 17:14:33,609 INFO [main] HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=edhuser ip=unknown-ip-addr cmd=get_tables: db=default pat=.*
2014-10-03 17:14:33,612 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=task.DDL.Stage-0 start=1412327673605 end=1412327673612 duration=7>
2014-10-03 17:14:33,613 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=runTasks start=1412327673605 end=1412327673613 duration=8>
2014-10-03 17:14:33,613 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=Driver.execute start=1412327673605 end=1412327673613 duration=8>
2014-10-03 17:14:33,613 INFO [main] ql.Driver (SessionState.java:printInfo(410)) - OK
2014-10-03 17:14:33,613 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=releaseLocks>
2014-10-03 17:14:33,613 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=releaseLocks start=1412327673613 end=1412327673613 duration=0>
2014-10-03 17:14:33,613 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=Driver.run start=1412327673600 end=1412327673613 duration=13>
2014-10-03 17:14:33,614 INFO [main] mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2014-10-03 17:14:33,615 INFO [main] ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=releaseLocks>
2014-10-03 17:14:33,615 INFO [main] ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=releaseLocks start=1412327673615 end=1412327673615 duration=0>
res4: org.apache.spark.sql.SchemaRDD =
SchemaRDD[3] at RDD at SchemaRDD.scala:103
== Query Plan ==
<Native command: executed by Hive>
scala>
On 3 Oct, 2014, at 4:52 pm, Michael Armbrust <mi...@databricks.com> wrote:
> Are you running master? There was briefly a regression here that is hopefully fixed by spark#2635.
>
> On Fri, Oct 3, 2014 at 1:43 AM, Kevin Paul <ke...@gmail.com> wrote:
> Hi all, I tried to launch my application with spark-submit, the command I use is:
>
> bin/spark-submit --class ${MY_CLASS} --jars ${MY_JARS} --master local myApplicationJar.jar
>
> I've buillt spark with SPARK_HIVE=true, and was able to start HiveContext, and was able to run command like,
> hiveContext.sql("select * from myTable")
> or hiveContext.sql("select count(*) from myTable")
> myTable is a table on my hive database. However, when I run the command: hiveContext.sql("show tables"), I got the following error:
>
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.Driver.validateConfVariables(Driver.java:1057)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:948)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:298)
> at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:272)
> at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
> at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
> at org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38)
> at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360)
> at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360)
>
> Thanks,
> Kelvin Paul
>
Re: SparkSQL on Hive error
Posted by Michael Armbrust <mi...@databricks.com>.
Are you running master? There was briefly a regression here that is
hopefully fixed by spark#2635 <https://github.com/apache/spark/pull/2635>.
On Fri, Oct 3, 2014 at 1:43 AM, Kevin Paul <ke...@gmail.com> wrote:
> Hi all, I tried to launch my application with spark-submit, the command I
> use is:
>
> bin/spark-submit --class ${MY_CLASS} --jars ${MY_JARS} --master local
> myApplicationJar.jar
>
> I've buillt spark with SPARK_HIVE=true, and was able to start HiveContext,
> and was able to run command like,
> hiveContext.sql("select * from myTable")
> or hiveContext.sql("select count(*) from myTable")
> myTable is a table on my hive database. However, when I run the command:
> hiveContext.sql("show tables"), I got the following error:
>
> java.lang.NullPointerException
>
> at org.apache.hadoop.hive.ql.Driver.validateConfVariables(Driver.java:1057)
>
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:948)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
>
> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:298)
>
> at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:272)
>
> at
> org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
>
> at
> org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
>
> at
> org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38)
>
> at
> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360)
>
> at
> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360)
>
>
> Thanks,
> Kelvin Paul
>