You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Ruslan Dautkhanov <da...@gmail.com> on 2016/11/24 20:02:41 UTC
sqlContext can't find tables, while HiveContext can
Problem 1, with sqlContext)
Spark 1.6
CDH 5.8.3
Zeppelin 0.6.2
Running
> sqlCtx = SQLContext(sc)
> sqlCtx.sql('select * from marketview.spend_dim')
shows exception "Table not found" .
The same runs find when using hiveContext.
See full stack in [1]
The same stack in the log file [2].
I probably wouldn't send this message seeking for your help,
but using hiveContext gives its own problems.
Any ideas why would sparkContext not see that table?
Problem 2, with HiveContext)
The other problem with hiveContext is brought up in another email
chain. Getting
You must *build Spark with Hive*. Export 'SPARK_HIVE=true'
The wierd part with this hiveContext problem - is that only happens
on second time we try to run a paragraph (and any consequative runs).
First time Zeppelin starts, I can see the same paragraph runs fine.
Zeppelin somehow corrupts its internal state after first run?
We use Jupyter notebooks without this problems in the same envrionment.
It might be something how Zeppelin was compiled?
This is how Zeppelin was built:
/opt/maven/maven-latest/bin/mvn clean package -DskipTests -Pspark-1.6
-Ppyspark -Dhadoop.version=2.6.0-cdh5.8.3 -Phadoop-2.6 -Pyarn -Pvendor-repo
-Pscala-2.10 -e
Any help will be greatly appreciated.
You see, I send this meesage on Thanksgiving, so it's an important problem
:-)
Happy Thanksgiving everyone! (if you celebrate it)
[1]
> Traceback (most recent call last):
> File "/tmp/zeppelin_pyspark-8000586427786928449.py", line 267, in <module>
> raise Exception(traceback.format_exc())
> Exception: Traceback (most recent call last):
> File "/tmp/zeppelin_pyspark-8000586427786928449.py", line 265, in <module>
> exec(code)
> File "<stdin>", line 2, in <module>
> File "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/context.py",
> line 580, in sql
> return DataFrame(self._ssql_ctx.sql(sqlQuery), self)
> File
> "/opt/cloudera/parcels/CDH/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py",
> line 813, in __call__
> answer, self.gateway_client, self.target_id, self.name)
> File "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/utils.py",
> line 51, in deco
> raise AnalysisException(s.split(': ', 1)[1], stackTrace)
> AnalysisException: u'Table not found: `marketview`.`spend_dim`;'
[2]
ERROR [2016-11-24 00:18:34,579] ({pool-2-thread-5}
> SparkSqlInterpreter.java[interpret]:120) - Invocation target exception
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.zeppelin.spark.SparkSqlInterpreter.interpret(SparkSqlInterpreter.java:115)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
> at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.sql.AnalysisException: Table not found:
> `marketview`.`mv_update_2016q1`;
> at
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:54)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:50)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:121)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:120)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:120)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:120)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:50)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:44)
> at
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34)
> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817)
> ... 16 more
> INFO [2016-11-24 00:18:34,581] ({pool-2-thread-5}
> SchedulerFactory.java[jobFinished]:137) - Job
> remoteInterpretJob_1479971914506 finished by scheduler
> org.apache.zeppelin.spark.SparkInterpreter866606804
Thank you,
Ruslan
Re: sqlContext can't find tables, while HiveContext can
Posted by Jeff Zhang <zj...@gmail.com>.
First you need to figure out where is the table. Is this table registered
in spark sql code or a hive table ?
If it is hive table, then check whether you put hive-site.xml on classpath
and configure metastore uri correctly in hive-site.xml.
look at the interpreter log to see which metastore it is using.
Ruslan Dautkhanov <da...@gmail.com>于2016年11月25日周五 上午4:04写道：
> Problem 1, with sqlContext)
> Spark 1.6
> CDH 5.8.3
> Zeppelin 0.6.2
>
> Running
>
> sqlCtx = SQLContext(sc)
> sqlCtx.sql('select * from marketview.spend_dim')
>
>
> shows exception "Table not found" .
> The same runs find when using hiveContext.
> See full stack in [1]
> The same stack in the log file [2].
>
> I probably wouldn't send this message seeking for your help,
> but using hiveContext gives its own problems.
> Any ideas why would sparkContext not see that table?
>
> Problem 2, with HiveContext)
> The other problem with hiveContext is brought up in another email
> chain. Getting
> You must *build Spark with Hive*. Export 'SPARK_HIVE=true'
> The wierd part with this hiveContext problem - is that only happens
> on second time we try to run a paragraph (and any consequative runs).
> First time Zeppelin starts, I can see the same paragraph runs fine.
> Zeppelin somehow corrupts its internal state after first run?
>
> We use Jupyter notebooks without this problems in the same envrionment.
> It might be something how Zeppelin was compiled?
>
> This is how Zeppelin was built:
> /opt/maven/maven-latest/bin/mvn clean package -DskipTests -Pspark-1.6
> -Ppyspark -Dhadoop.version=2.6.0-cdh5.8.3 -Phadoop-2.6 -Pyarn -Pvendor-repo
> -Pscala-2.10 -e
>
> Any help will be greatly appreciated.
> You see, I send this meesage on Thanksgiving, so it's an important problem
> :-)
> Happy Thanksgiving everyone! (if you celebrate it)
>
>
> [1]
>
> Traceback (most recent call last):
> File "/tmp/zeppelin_pyspark-8000586427786928449.py", line 267, in <module>
> raise Exception(traceback.format_exc())
> Exception: Traceback (most recent call last):
> File "/tmp/zeppelin_pyspark-8000586427786928449.py", line 265, in <module>
> exec(code)
> File "<stdin>", line 2, in <module>
> File "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/context.py",
> line 580, in sql
> return DataFrame(self._ssql_ctx.sql(sqlQuery), self)
> File
> "/opt/cloudera/parcels/CDH/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py",
> line 813, in __call__
> answer, self.gateway_client, self.target_id, self.name)
> File "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/utils.py",
> line 51, in deco
> raise AnalysisException(s.split(': ', 1)[1], stackTrace)
> AnalysisException: u'Table not found: `marketview`.`spend_dim`;'
>
>
> [2]
>
> ERROR [2016-11-24 00:18:34,579] ({pool-2-thread-5}
> SparkSqlInterpreter.java[interpret]:120) - Invocation target exception
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.zeppelin.spark.SparkSqlInterpreter.interpret(SparkSqlInterpreter.java:115)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
> at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.sql.AnalysisException: Table not found:
> `marketview`.`mv_update_2016q1`;
> at
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:54)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:50)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:121)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:120)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:120)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:120)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:50)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:44)
> at
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34)
> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817)
> ... 16 more
> INFO [2016-11-24 00:18:34,581] ({pool-2-thread-5}
> SchedulerFactory.java[jobFinished]:137) - Job
> remoteInterpretJob_1479971914506 finished by scheduler
> org.apache.zeppelin.spark.SparkInterpreter866606804
>
>
>
> Thank you,
> Ruslan
>