You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Todd Nist <ts...@gmail.com> on 2015/04/06 20:37:59 UTC
Spark SQL Parquet as External table - 1.3.x HiveMetastoreType now hidden
In 1.2.1 of I was persisting a set of parquet files as a table for use by
spark-sql cli later on. There was a post here
<http://apache-spark-user-list.1001560.n3.nabble.com/persist-table-schema-in-spark-sql-tt16297.html#a16311>
by
Mchael Armbrust that provide a nice little helper method for dealing with
this:
/**
* Sugar for creating a Hive external table from a parquet path.
*/def createParquetTable(name: String, file: String): Unit = {
import org.apache.spark.sql.hive.HiveMetastoreTypes
val rdd = parquetFile(file)
val schema = rdd.schema.fields.map(f => s"${f.name}
${HiveMetastoreTypes.toMetastoreType(f.dataType)}").mkString(",\n")
val ddl = s"""
|CREATE EXTERNAL TABLE $name (
| $schema
|)
|ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
|STORED AS INPUTFORMAT 'parquet.hive.DeprecatedParquetInputFormat'
|OUTPUTFORMAT 'parquet.hive.DeprecatedParquetOutputFormat'
|LOCATION '$file'""".stripMargin
sql(ddl)
setConf("spark.sql.hive.convertMetastoreParquet", "true")
}
In migrating to 1.3.x I see that the spark.sql.hive.convertMetastoreParquet
is no longer public, so the above no longer works.
I can define a helper method that wraps the HiveMetastoreTypes something
like:
package org.apache.spark.sql.hive
import org.apache.spark.sql.types.DataType
/**
* Helper to expose HiveMetastoreTypes hidden by Spark. It is created
in this name space to make it accessible.
*/
object HiveTypeHelper {
def toDataType(metastoreType: String): DataType =
HiveMetastoreTypes.toDataType(metastoreType)
def toMetastoreType(dataType: DataType): String =
HiveMetastoreTypes.toMetastoreType(dataType)
}
While this will work, is there a better way to achieve this under 1.3.x?
TIA for the assistance.
-Todd
Re: Spark SQL Parquet as External table - 1.3.x HiveMetastoreType now hidden
Posted by Michael Armbrust <mi...@databricks.com>.
>
> Here is the stack trace. The first part shows the log when the session is
> started in Tableau. It is using the init sql option on the data
> connection to create theTEMPORARY table myNodeTable.
>
Ah, I see. thanks for providing the error. The problem here is that
temporary tables do not exist in a database. They are visible no matter
what the current database is. Tableau is asking for
default.temporaryTable, which does not exist.
Re: Spark SQL Parquet as External table - 1.3.x HiveMetastoreType now hidden
Posted by Todd Nist <ts...@gmail.com>.
I can see how disabling the caching would be of value especially if one
needs to see the data in a near real-time fashion being updated, deleted or
created. I'm not sure internally what impact that would have on
performance.
For now, I will try just starting up my own HiveThirftServer(context),
passing in a common sql context and see how that goes.
As for the exception, here it is.
Here is the stack trace. The first part shows the log when the session is
started in Tableau. It is using the init sql option on the data connection
to create theTEMPORARY table myNodeTable.
15/04/06 18:04:38 INFO thrift.ThriftCLIService: Client protocol
version: HIVE_CLI_SERVICE_PROTOCOL_V1
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO thriftserver.SparkExecuteStatementOperation:
Running query 'use `default`'
15/04/06 18:04:38 INFO parse.ParseDriver: Parsing command: use `default`
15/04/06 18:04:38 INFO parse.ParseDriver: Parse Completed
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=Driver.run
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Concurrency mode is disabled, not
creating a lock manager
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=compile
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=parse
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO parse.ParseDriver: Parsing command: use `default`
15/04/06 18:04:38 INFO parse.ParseDriver: Parse Completed
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=parse
start=1428357878679 end=1428357878680 duration=1
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Semantic Analysis Completed
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG
method=semanticAnalyze start=1428357878680 end=1428357878680
duration=0 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Returning Hive schema:
Schema(fieldSchemas:null, properties:null)
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=compile
start=1428357878679 end=1428357878681 duration=2
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=Driver.execute
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Starting command: use `default`
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit
start=1428357878679 end=1428357878681 duration=2
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=runTasks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG
method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=runTasks
start=1428357878681 end=1428357878713 duration=32
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=Driver.execute
start=1428357878681 end=1428357878713 duration=32
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: OK
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=releaseLocks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=releaseLocks
start=1428357878714 end=1428357878714 duration=0
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=Driver.run
start=1428357878679 end=1428357878714 duration=35
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO thrift.ThriftCLIService: Client protocol
version: HIVE_CLI_SERVICE_PROTOCOL_V1
15/04/06 18:04:38 INFO hive.metastore: Trying to connect to metastore
with URI thrift://hadoopdev03.opsdatastore.com:9083
15/04/06 18:04:38 INFO hive.metastore: Connected to metastore.
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO thriftserver.SparkExecuteStatementOperation:
Running query 'use `default`'
15/04/06 18:04:38 INFO parse.ParseDriver: Parsing command: use `default`
15/04/06 18:04:38 INFO parse.ParseDriver: Parse Completed
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=Driver.run
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Concurrency mode is disabled, not
creating a lock manager
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=compile
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=parse
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO parse.ParseDriver: Parsing command: use `default`
15/04/06 18:04:38 INFO parse.ParseDriver: Parse Completed
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=parse
start=1428357878781 end=1428357878782 duration=1
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Semantic Analysis Completed
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG
method=semanticAnalyze start=1428357878782 end=1428357878782
duration=0 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Returning Hive schema:
Schema(fieldSchemas:null, properties:null)
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=compile
start=1428357878781 end=1428357878783 duration=2
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=Driver.execute
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Starting command: use `default`
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit
start=1428357878781 end=1428357878783 duration=2
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=runTasks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG
method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=runTasks
start=1428357878783 end=1428357878819 duration=36
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=Driver.execute
start=1428357878783 end=1428357878820 duration=37
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: OK
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=releaseLocks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=releaseLocks
start=1428357878820 end=1428357878820 duration=0
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=Driver.run
start=1428357878781 end=1428357878820 duration=39
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO thriftserver.SparkExecuteStatementOperation:
Running query 'set -v'
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO thriftserver.SparkExecuteStatementOperation:
Result Schema: List(#20)
15/04/06 18:04:38 INFO thriftserver.SparkExecuteStatementOperation:
Result Schema: List(#20)
15/04/06 18:04:38 INFO thrift.ThriftCLIService: Client protocol
version: HIVE_CLI_SERVICE_PROTOCOL_V6
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO thriftserver.SparkExecuteStatementOperation:
Running query 'use `default`'
15/04/06 18:04:38 INFO parse.ParseDriver: Parsing command: use `default`
15/04/06 18:04:38 INFO parse.ParseDriver: Parse Completed
15/04/06 18:04:38 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=Driver.run
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Concurrency mode is disabled, not
creating a lock manager
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=compile
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=parse
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO parse.ParseDriver: Parsing command: use `default`
15/04/06 18:04:38 INFO parse.ParseDriver: Parse Completed
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=parse
start=1428357878942 end=1428357878942 duration=0
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Semantic Analysis Completed
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG
method=semanticAnalyze start=1428357878942 end=1428357878943
duration=1 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Returning Hive schema:
Schema(fieldSchemas:null, properties:null)
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=compile
start=1428357878941 end=1428357878943 duration=2
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=Driver.execute
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: Starting command: use `default`
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit
start=1428357878941 end=1428357878944 duration=3
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=runTasks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG
method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=runTasks
start=1428357878944 end=1428357878966 duration=22
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=Driver.execute
start=1428357878943 end=1428357878967 duration=24
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO ql.Driver: OK
15/04/06 18:04:38 INFO log.PerfLogger: <PERFLOG method=releaseLocks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=releaseLocks
start=1428357878967 end=1428357878967 duration=0
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:38 INFO log.PerfLogger: </PERFLOG method=Driver.run
start=1428357878941 end=1428357878967 duration=26
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:04:39 INFO thriftserver.SparkExecuteStatementOperation:
Running query 'CREATE TEMPORARY TABLE myNodeTest
USING org.apache.spark.sql.parquet
OPTIONS (
path '/user/data/parquet/demo-data/nodes.parquet'
) '
15/04/06 18:04:39 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:05:06 INFO thrift.ThriftCLIService: Client protocol
version: HIVE_CLI_SERVICE_PROTOCOL_V6
15/04/06 18:05:06 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:05:06 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:05:06 INFO thriftserver.SparkExecuteStatementOperation:
Running query 'use `default`'
15/04/06 18:05:06 INFO parse.ParseDriver: Parsing command: use `default`
15/04/06 18:05:06 INFO parse.ParseDriver: Parse Completed
15/04/06 18:05:06 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=Driver.run
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO ql.Driver: Concurrency mode is disabled, not
creating a lock manager
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=compile
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=parse
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO parse.ParseDriver: Parsing command: use `default`
15/04/06 18:05:06 INFO parse.ParseDriver: Parse Completed
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=parse
start=1428357906623 end=1428357906624 duration=1
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO ql.Driver: Semantic Analysis Completed
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG
method=semanticAnalyze start=1428357906625 end=1428357906625
duration=0 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO ql.Driver: Returning Hive schema:
Schema(fieldSchemas:null, properties:null)
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=compile
start=1428357906623 end=1428357906626 duration=3
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=Driver.execute
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO ql.Driver: Starting command: use `default`
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit
start=1428357906622 end=1428357906627 duration=5
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=runTasks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG
method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=runTasks
start=1428357906627 end=1428357906652 duration=25
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=Driver.execute
start=1428357906626 end=1428357906653 duration=27
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO ql.Driver: OK
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=releaseLocks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=releaseLocks
start=1428357906654 end=1428357906654 duration=0
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=Driver.run
start=1428357906622 end=1428357906654 duration=32
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO thriftserver.SparkExecuteStatementOperation:
Running query 'SHOW SCHEMAS'
15/04/06 18:05:06 INFO parse.ParseDriver: Parsing command: SHOW SCHEMAS
15/04/06 18:05:06 INFO parse.ParseDriver: Parse Completed
15/04/06 18:05:06 INFO session.SessionState: No Tez session required
at this point. hive.execution.engine=mr.
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=Driver.run
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO ql.Driver: Concurrency mode is disabled, not
creating a lock manager
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=compile
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=parse
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO parse.ParseDriver: Parsing command: SHOW SCHEMAS
15/04/06 18:05:06 INFO parse.ParseDriver: Parse Completed
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=parse
start=1428357906887 end=1428357906888 duration=1
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO ql.Driver: Semantic Analysis Completed
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG
method=semanticAnalyze start=1428357906888 end=1428357906912
duration=24 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO exec.ListSinkOperator: Initializing Self 0 OP
15/04/06 18:05:06 INFO exec.ListSinkOperator: Operator 0 OP initialized
15/04/06 18:05:06 INFO exec.ListSinkOperator: Initialization Done 0 OP
15/04/06 18:05:06 INFO ql.Driver: Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:database_name, type:string,
comment:from deserializer)], properties:null)
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=compile
start=1428357906886 end=1428357906915 duration=29
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=Driver.execute
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO ql.Driver: Starting command: SHOW SCHEMAS
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit
start=1428357906886 end=1428357906916 duration=30
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=runTasks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG
method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO exec.DDLTask: results : 1
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=runTasks
start=1428357906917 end=1428357906932 duration=15
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=Driver.execute
start=1428357906916 end=1428357906933 duration=17
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO ql.Driver: OK
15/04/06 18:05:06 INFO log.PerfLogger: <PERFLOG method=releaseLocks
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=releaseLocks
start=1428357906933 end=1428357906934 duration=1
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO log.PerfLogger: </PERFLOG method=Driver.run
start=1428357906886 end=1428357906934 duration=48
from=org.apache.hadoop.hive.ql.Driver>
15/04/06 18:05:06 INFO mapred.FileInputFormat: Total input paths to process : 1
15/04/06 18:05:06 INFO exec.ListSinkOperator: 0 finished. closing...
15/04/06 18:05:06 INFO exec.ListSinkOperator: 0 Close done
15/04/06 18:05:07 INFO thriftserver.SparkExecuteStatementOperation:
Result Schema: List(result#498)
15/04/06 18:05:07 INFO thriftserver.SparkExecuteStatementOperation:
Result Schema: List(result#501)
15/04/06 18:05:07 INFO thriftserver.SparkExecuteStatementOperation:
Result Schema: List(result#504)
The table shows up under the Table section in Tableau. When I select
myNodeTable and click I get the following:
And the log shows the following on the ThriftServer:
15/04/06 18:05:16 INFO thrift.ThriftCLIService: Client protocol
version: HIVE_CLI_SERVICE_PROTOCOL_V615/04/06 18:05:16 INFO
session.SessionState: No Tez session required at this point.
hive.execution.engine=mr.15/04/06 18:05:16 INFO session.SessionState:
No Tez session required at this point.
hive.execution.engine=mr.15/04/06 18:05:16 INFO
thriftserver.SparkExecuteStatementOperation: Running query 'use
`default`'15/04/06 18:05:17 INFO parse.ParseDriver: Parsing command:
use `default`15/04/06 18:05:17 INFO parse.ParseDriver: Parse
Completed15/04/06 18:05:17 INFO session.SessionState: No Tez session
required at this point. hive.execution.engine=mr.15/04/06 18:05:17
INFO log.PerfLogger: <PERFLOG method=Driver.run
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
log.PerfLogger: <PERFLOG method=TimeToSubmit
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
ql.Driver: Concurrency mode is disabled, not creating a lock
manager15/04/06 18:05:17 INFO log.PerfLogger: <PERFLOG method=compile
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
log.PerfLogger: <PERFLOG method=parse
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
parse.ParseDriver: Parsing command: use `default`15/04/06 18:05:17
INFO parse.ParseDriver: Parse Completed15/04/06 18:05:17 INFO
log.PerfLogger: </PERFLOG method=parse start=1428357917033
end=1428357917034 duration=1
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
log.PerfLogger: <PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
ql.Driver: Semantic Analysis Completed15/04/06 18:05:17 INFO
log.PerfLogger: </PERFLOG method=semanticAnalyze start=1428357917035
end=1428357917035 duration=0
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
ql.Driver: Returning Hive schema: Schema(fieldSchemas:null,
properties:null)15/04/06 18:05:17 INFO log.PerfLogger: </PERFLOG
method=compile start=1428357917032 end=1428357917036 duration=4
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
log.PerfLogger: <PERFLOG method=Driver.execute
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
ql.Driver: Starting command: use `default`15/04/06 18:05:17 INFO
log.PerfLogger: </PERFLOG method=TimeToSubmit start=1428357917031
end=1428357917037 duration=6
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
log.PerfLogger: <PERFLOG method=runTasks
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
log.PerfLogger: <PERFLOG method=task.DDL.Stage-0
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
log.PerfLogger: </PERFLOG method=runTasks start=1428357917038
end=1428357917064 duration=26
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
log.PerfLogger: </PERFLOG method=Driver.execute start=1428357917037
end=1428357917064 duration=27
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
ql.Driver: OK15/04/06 18:05:17 INFO log.PerfLogger: <PERFLOG
method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>15/04/06
18:05:17 INFO log.PerfLogger: </PERFLOG method=releaseLocks
start=1428357917065 end=1428357917065 duration=0
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
log.PerfLogger: </PERFLOG method=Driver.run start=1428357917031
end=1428357917065 duration=34
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:17 INFO
thriftserver.SparkExecuteStatementOperation: Running query 'SHOW
TABLES IN `default`'15/04/06 18:05:17 INFO session.SessionState: No
Tez session required at this point. hive.execution.engine=mr.15/04/06
18:05:17 INFO thriftserver.SparkExecuteStatementOperation: Result
Schema: List(tableName#517, isTemporary#518)15/04/06 18:05:17 INFO
thriftserver.SparkExecuteStatementOperation: Result Schema:
List(tableName#517, isTemporary#518)15/04/06 18:05:17 INFO
thriftserver.SparkExecuteStatementOperation: Result Schema:
List(tableName#517, isTemporary#518)15/04/06 18:05:28 INFO
thrift.ThriftCLIService: Client protocol version:
HIVE_CLI_SERVICE_PROTOCOL_V615/04/06 18:05:28 INFO
session.SessionState: No Tez session required at this point.
hive.execution.engine=mr.15/04/06 18:05:28 INFO session.SessionState:
No Tez session required at this point.
hive.execution.engine=mr.15/04/06 18:05:28 INFO
thriftserver.SparkExecuteStatementOperation: Running query 'use
`default`'15/04/06 18:05:29 INFO parse.ParseDriver: Parsing command:
use `default`15/04/06 18:05:29 INFO parse.ParseDriver: Parse
Completed15/04/06 18:05:29 INFO session.SessionState: No Tez session
required at this point. hive.execution.engine=mr.15/04/06 18:05:29
INFO log.PerfLogger: <PERFLOG method=Driver.run
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
log.PerfLogger: <PERFLOG method=TimeToSubmit
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
ql.Driver: Concurrency mode is disabled, not creating a lock
manager15/04/06 18:05:29 INFO log.PerfLogger: <PERFLOG method=compile
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
log.PerfLogger: <PERFLOG method=parse
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
parse.ParseDriver: Parsing command: use `default`15/04/06 18:05:29
INFO parse.ParseDriver: Parse Completed15/04/06 18:05:29 INFO
log.PerfLogger: </PERFLOG method=parse start=1428357929015
end=1428357929015 duration=0
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
log.PerfLogger: <PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
ql.Driver: Semantic Analysis Completed15/04/06 18:05:29 INFO
log.PerfLogger: </PERFLOG method=semanticAnalyze start=1428357929015
end=1428357929016 duration=1
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
ql.Driver: Returning Hive schema: Schema(fieldSchemas:null,
properties:null)15/04/06 18:05:29 INFO log.PerfLogger: </PERFLOG
method=compile start=1428357929014 end=1428357929016 duration=2
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
log.PerfLogger: <PERFLOG method=Driver.execute
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
ql.Driver: Starting command: use `default`15/04/06 18:05:29 INFO
log.PerfLogger: </PERFLOG method=TimeToSubmit start=1428357929014
end=1428357929017 duration=3
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
log.PerfLogger: <PERFLOG method=runTasks
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
log.PerfLogger: <PERFLOG method=task.DDL.Stage-0
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
log.PerfLogger: </PERFLOG method=runTasks start=1428357929017
end=1428357929041 duration=24
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
log.PerfLogger: </PERFLOG method=Driver.execute start=1428357929017
end=1428357929042 duration=25
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
ql.Driver: OK15/04/06 18:05:29 INFO log.PerfLogger: <PERFLOG
method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>15/04/06
18:05:29 INFO log.PerfLogger: </PERFLOG method=releaseLocks
start=1428357929042 end=1428357929043 duration=1
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
log.PerfLogger: </PERFLOG method=Driver.run start=1428357929014
end=1428357929043 duration=29
from=org.apache.hadoop.hive.ql.Driver>15/04/06 18:05:29 INFO
thriftserver.SparkExecuteStatementOperation: Running query 'SELECT *
FROM `default`.`mynodetest`
LIMIT 0'15/04/06 18:05:29 INFO parse.ParseDriver: Parsing command: SELECT *
FROM `default`.`mynodetest`
LIMIT 015/04/06 18:05:29 INFO parse.ParseDriver: Parse
Completed15/04/06 18:05:29 INFO session.SessionState: No Tez session
required at this point. hive.execution.engine=mr.15/04/06 18:05:29
ERROR metadata.Hive: NoSuchObjectException(message:default.mynodetest
table not found)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29338)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29306)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result.read(ThriftHiveMetastore.java:29237)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1036)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1022)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:997)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at com.sun.proxy.$Proxy27.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:976)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:950)
at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:180)
at org.apache.spark.sql.hive.HiveContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(HiveContext.scala:252)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:161)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:161)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:161)
at org.apache.spark.sql.hive.HiveContext$$anon$2.lookupRelation(HiveContext.scala:252)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:175)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$6.applyOrElse(Analyzer.scala:187)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$6.applyOrElse(Analyzer.scala:182)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:187)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:187)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:50)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:186)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:236)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:236)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:177)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:182)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:172)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:61)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:59)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:59)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:51)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:51)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:1071)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:1071)
at org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:1069)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:92)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:164)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy28.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:344)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15/04/06 18:05:29 ERROR thriftserver.SparkExecuteStatementOperation:
Error executing query:
org.apache.spark.sql.AnalysisException: no such table List(default,
mynodetest); line 2 pos 15
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:178)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$6.applyOrElse(Analyzer.scala:187)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$6.applyOrElse(Analyzer.scala:182)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:187)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:187)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:50)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:186)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:236)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:236)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:177)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:182)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:172)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:61)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:59)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:59)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:51)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:51)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:1071)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:1071)
at org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:1069)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:92)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:164)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy28.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:344)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)15/04/06 18:05:29 WARN
thrift.ThriftCLIService: Error executing statement:
org.apache.hive.service.cli.HiveSQLException:
org.apache.spark.sql.AnalysisException: no such table List(default,
mynodetest); line 2 pos 15
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:189)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy28.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:344)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
-Todd
On Mon, Apr 6, 2015 at 5:40 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> Which caused some issues with the BI tool I'm using, Tableau. It would
>> show the table under tables, but when selected would throw an exception.
>> Removing the "TEMPORARY" appears to address that problem.
>>
> Hmm, can you provide the exception? We made some changes in 1.3
> specifically to allow BI tools to see temporary tables, so I'd like to know
> if this is not working correctly.
>
>
>> I do have one remaining concern. If one wants to reflect realtime
>> updates, using this approach, will they show up? I wold think I would need
>> to issue a "REFRESH TABLE tableName" when an update occurs; is that
>> correct? If I have a streaming job that is updating a set of tables, and
>> issues the REFRESH TABLE as the final step in the pipeline do you know if
>> the changes will be reflected in the referenced data by Tableau session? I
>> need to try that next, just thought you or someone on the forum may know
>> answer.
>>
> The cached metadata (and any refreshing that is done) is handled on a per
> SQLContext basis. So, if your streaming job is running on the same SQL
> context as your JDBC server then this would work. Otherwise you will need
> to call it manually from the context you are reading from. We could also
> consider adding an option to disable caching if you think that would be
> useful.
>
Re: Spark SQL Parquet as External table - 1.3.x HiveMetastoreType now hidden
Posted by Michael Armbrust <mi...@databricks.com>.
>
> Which caused some issues with the BI tool I'm using, Tableau. It would
> show the table under tables, but when selected would throw an exception.
> Removing the "TEMPORARY" appears to address that problem.
>
Hmm, can you provide the exception? We made some changes in 1.3
specifically to allow BI tools to see temporary tables, so I'd like to know
if this is not working correctly.
> I do have one remaining concern. If one wants to reflect realtime
> updates, using this approach, will they show up? I wold think I would need
> to issue a "REFRESH TABLE tableName" when an update occurs; is that
> correct? If I have a streaming job that is updating a set of tables, and
> issues the REFRESH TABLE as the final step in the pipeline do you know if
> the changes will be reflected in the referenced data by Tableau session? I
> need to try that next, just thought you or someone on the forum may know
> answer.
>
The cached metadata (and any refreshing that is done) is handled on a per
SQLContext basis. So, if your streaming job is running on the same SQL
context as your JDBC server then this would work. Otherwise you will need
to call it manually from the context you are reading from. We could also
consider adding an option to disable caching if you think that would be
useful.
Re: Spark SQL Parquet as External table - 1.3.x HiveMetastoreType now hidden
Posted by Todd Nist <ts...@gmail.com>.
Hi Michael,
Yes that was a typo. Thanks for the pointer, it works now.
I had tried that but had one thing off. I had
CREATE TEMPORARY TABLE tableName
USING parquet
OPTIONS (
path '/path/to/file'
)
Which caused some issues with the BI tool I'm using, Tableau. It would
show the table under tables, but when selected would throw an exception.
Removing the "TEMPORARY" appears to address that problem.
I do have one remaining concern. If one wants to reflect realtime updates,
using this approach, will they show up? I wold think I would need to issue
a "REFRESH TABLE tableName" when an update occurs; is that correct? If I
have a streaming job that is updating a set of tables, and issues the
REFRESH TABLE as the final step in the pipeline do you know if the changes
will be reflected in the referenced data by Tableau session? I need to try
that next, just thought you or someone on the forum may know answer.
Thanks again.
-Todd
On Mon, Apr 6, 2015 at 4:36 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> Hey Todd,
>
> In migrating to 1.3.x I see that the
>> spark.sql.hive.convertMetastoreParquet is no longer public, so the above no
>> longer works.
>
>
> This was probably just a typo, but to be clear,
> spark.sql.hive.convertMetastoreParquet is still a supported option and
> should work. You are correct that the HiveMetastoreTypes class is now
> private (we made a lot of stuff private starting with 1.3 (and the removal
> of "alpha") since we are now promising binary compatibility for public
> APIs. Your hack seems reasonable, but I'll caution this is not a stable
> public API so could break with future upgrades.
>
> While this will work, is there a better way to achieve this under 1.3.x?
>
>
> If you are only looking for the ability to read this data with Spark SQL
> (and not Hive) I suggest you look at the Data Sources API syntax for
> creating tables. You don't need to specify the schema at all for self
> describing formats like parquet.
>
> CREATE TABLE tableName
> USING parquet
> OPTIONS (
> path '/path/to/file'
> )
>
> Michael
>
>
> On Mon, Apr 6, 2015 at 11:37 AM, Todd Nist <ts...@gmail.com> wrote:
>
>> In 1.2.1 of I was persisting a set of parquet files as a table for use by
>> spark-sql cli later on. There was a post here
>> <http://apache-spark-user-list.1001560.n3.nabble.com/persist-table-schema-in-spark-sql-tt16297.html#a16311> by
>> Mchael Armbrust that provide a nice little helper method for dealing with
>> this:
>>
>> /**
>> * Sugar for creating a Hive external table from a parquet path.
>> */def createParquetTable(name: String, file: String): Unit = {
>> import org.apache.spark.sql.hive.HiveMetastoreTypes
>>
>> val rdd = parquetFile(file)
>> val schema = rdd.schema.fields.map(f => s"${f.name} ${HiveMetastoreTypes.toMetastoreType(f.dataType)}").mkString(",\n")
>> val ddl = s"""
>> |CREATE EXTERNAL TABLE $name (
>> | $schema
>> |)
>> |ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
>> |STORED AS INPUTFORMAT 'parquet.hive.DeprecatedParquetInputFormat'
>> |OUTPUTFORMAT 'parquet.hive.DeprecatedParquetOutputFormat'
>> |LOCATION '$file'""".stripMargin
>> sql(ddl)
>> setConf("spark.sql.hive.convertMetastoreParquet", "true")
>> }
>>
>> In migrating to 1.3.x I see that the
>> spark.sql.hive.convertMetastoreParquet is no longer public, so the above no
>> longer works.
>>
>> I can define a helper method that wraps the HiveMetastoreTypes something
>> like:
>>
>> package org.apache.spark.sql.hive
>> import org.apache.spark.sql.types.DataType
>>
>> /**
>> * Helper to expose HiveMetastoreTypes hidden by Spark. It is created in this name space to make it accessible.
>> */
>> object HiveTypeHelper {
>> def toDataType(metastoreType: String): DataType = HiveMetastoreTypes.toDataType(metastoreType)
>> def toMetastoreType(dataType: DataType): String = HiveMetastoreTypes.toMetastoreType(dataType)
>> }
>>
>> While this will work, is there a better way to achieve this under 1.3.x?
>>
>> TIA for the assistance.
>>
>> -Todd
>>
>
>
Re: Spark SQL Parquet as External table - 1.3.x HiveMetastoreType now hidden
Posted by Michael Armbrust <mi...@databricks.com>.
Hey Todd,
In migrating to 1.3.x I see that the spark.sql.hive.convertMetastoreParquet
> is no longer public, so the above no longer works.
This was probably just a typo, but to be clear,
spark.sql.hive.convertMetastoreParquet is still a supported option and
should work. You are correct that the HiveMetastoreTypes class is now
private (we made a lot of stuff private starting with 1.3 (and the removal
of "alpha") since we are now promising binary compatibility for public
APIs. Your hack seems reasonable, but I'll caution this is not a stable
public API so could break with future upgrades.
While this will work, is there a better way to achieve this under 1.3.x?
If you are only looking for the ability to read this data with Spark SQL
(and not Hive) I suggest you look at the Data Sources API syntax for
creating tables. You don't need to specify the schema at all for self
describing formats like parquet.
CREATE TABLE tableName
USING parquet
OPTIONS (
path '/path/to/file'
)
Michael
On Mon, Apr 6, 2015 at 11:37 AM, Todd Nist <ts...@gmail.com> wrote:
> In 1.2.1 of I was persisting a set of parquet files as a table for use by
> spark-sql cli later on. There was a post here
> <http://apache-spark-user-list.1001560.n3.nabble.com/persist-table-schema-in-spark-sql-tt16297.html#a16311> by
> Mchael Armbrust that provide a nice little helper method for dealing with
> this:
>
> /**
> * Sugar for creating a Hive external table from a parquet path.
> */def createParquetTable(name: String, file: String): Unit = {
> import org.apache.spark.sql.hive.HiveMetastoreTypes
>
> val rdd = parquetFile(file)
> val schema = rdd.schema.fields.map(f => s"${f.name} ${HiveMetastoreTypes.toMetastoreType(f.dataType)}").mkString(",\n")
> val ddl = s"""
> |CREATE EXTERNAL TABLE $name (
> | $schema
> |)
> |ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
> |STORED AS INPUTFORMAT 'parquet.hive.DeprecatedParquetInputFormat'
> |OUTPUTFORMAT 'parquet.hive.DeprecatedParquetOutputFormat'
> |LOCATION '$file'""".stripMargin
> sql(ddl)
> setConf("spark.sql.hive.convertMetastoreParquet", "true")
> }
>
> In migrating to 1.3.x I see that the
> spark.sql.hive.convertMetastoreParquet is no longer public, so the above no
> longer works.
>
> I can define a helper method that wraps the HiveMetastoreTypes something
> like:
>
> package org.apache.spark.sql.hive
> import org.apache.spark.sql.types.DataType
>
> /**
> * Helper to expose HiveMetastoreTypes hidden by Spark. It is created in this name space to make it accessible.
> */
> object HiveTypeHelper {
> def toDataType(metastoreType: String): DataType = HiveMetastoreTypes.toDataType(metastoreType)
> def toMetastoreType(dataType: DataType): String = HiveMetastoreTypes.toMetastoreType(dataType)
> }
>
> While this will work, is there a better way to achieve this under 1.3.x?
>
> TIA for the assistance.
>
> -Todd
>