You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by caiqiang <qi...@qq.com> on 2016/07/06 02:52:46 UTC
回复：scala> cc.sql(s"load data inpath 'hdfs:/
From the data loading log, I find this table is not a carbon table.
Need to create carbon table and load data again.
------------------ 原始邮件 ------------------
发件人: "祝威廉";<we...@qq.com>;
发送时间: 2016年7月5日(星期二) 下午4:17
收件人: "caiqiang"<qi...@qq.com>;
主题: scala> cc.sql(s"load data inpath 'hdfs:/
scala> cc.sql(s"load data inpath 'hdfs://cdncluster/tmp/csv-panda_quality_v1_20160625/part-00000.csv' into table williamtable02")
INFO 05-07 16:11:30,316 - main Query [LOAD DATA INPATH 'HDFS://CDNCLUSTER/TMP/CSV-PANDA_QUALITY_V1_20160625/PART-00000.CSV' INTO TABLE WILLIAMTABLE02]
INFO 05-07 16:11:30,379 - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:30,379 - <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:30,379 - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:30,411 - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:30,414 - Parsing command: load data inpath 'hdfs://cdncluster/tmp/csv-panda_quality_v1_20160625/part-00000.csv' into table williamtable02
INFO 05-07 16:11:31,203 - Parse Completed
INFO 05-07 16:11:31,204 - </PERFLOG method=parse start=1467706290411 end=1467706291204 duration=793 from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:31,206 - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:31,254 - 0: get_table : db=default tbl=williamtable02
INFO 05-07 16:11:31,254 - ugi=allwefantasy ip=unknown-ip-addr cmd=get_table : db=default tbl=williamtable02
INFO 05-07 16:11:31,572 - Semantic Analysis Completed
INFO 05-07 16:11:31,572 - </PERFLOG method=semanticAnalyze start=1467706291206 end=1467706291572 duration=366 from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:31,579 - Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO 05-07 16:11:31,579 - </PERFLOG method=compile start=1467706290379 end=1467706291579 duration=1200 from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:31,579 - Concurrency mode is disabled, not creating a lock manager
INFO 05-07 16:11:31,580 - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:31,580 - Starting command(queryId=allwefantasy_20160705161130_e7a86d50-fbcd-402f-90f7-36b97ab224ef): load data inpath 'hdfs://cdncluster/tmp/csv-panda_quality_v1_20160625/part-00000.csv' into table williamtable02
INFO 05-07 16:11:31,587 - </PERFLOG method=TimeToSubmit start=1467706290379 end=1467706291587 duration=1208 from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:31,587 - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:31,587 - <PERFLOG method=task.MOVE.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:31,588 - Starting task [Stage-0:MOVE] in serial mode
INFO 05-07 16:11:31,589 - Loading data to table default.williamtable02 from hdfs://cdncluster/tmp/csv-panda_quality_v1_20160625/part-00000.csv
INFO 05-07 16:11:31,589 - 0: get_table : db=default tbl=williamtable02
INFO 05-07 16:11:31,589 - ugi=allwefantasy ip=unknown-ip-addr cmd=get_table : db=default tbl=williamtable02
INFO 05-07 16:11:31,749 - 0: get_table : db=default tbl=williamtable02
INFO 05-07 16:11:31,749 - ugi=allwefantasy ip=unknown-ip-addr cmd=get_table : db=default tbl=williamtable02
INFO 05-07 16:11:31,942 - Renaming src: hdfs://cdncluster/tmp/csv-panda_quality_v1_20160625/part-00000.csv, dest: hdfs://cdncluster/user/hive/warehouse/williamtable02/part-00000.csv, Status:true
chgrp: changing ownership of 'hdfs://cdncluster/user/hive/warehouse/williamtable02/part-00000.csv': User does not belong to cdnuser
INFO 05-07 16:11:32,009 - 0: alter_table: db=default tbl=williamtable02 newtbl=williamtable02
INFO 05-07 16:11:32,009 - ugi=allwefantasy ip=unknown-ip-addr cmd=alter_table: db=default tbl=williamtable02 newtbl=williamtable02
INFO 05-07 16:11:32,285 - Updating table stats fast for williamtable02
INFO 05-07 16:11:32,285 - Updated size of table williamtable02 to 389088954
INFO 05-07 16:11:32,425 - <PERFLOG method=task.STATS.Stage-1 from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:32,426 - Starting task [Stage-1:STATS] in serial mode
INFO 05-07 16:11:32,426 - Executing stats task
INFO 05-07 16:11:32,427 - 0: get_table : db=default tbl=williamtable02
INFO 05-07 16:11:32,427 - ugi=allwefantasy ip=unknown-ip-addr cmd=get_table : db=default tbl=williamtable02
INFO 05-07 16:11:32,560 - 0: get_table : db=default tbl=williamtable02
INFO 05-07 16:11:32,561 - ugi=allwefantasy ip=unknown-ip-addr cmd=get_table : db=default tbl=williamtable02
INFO 05-07 16:11:32,706 - 0: alter_table: db=default tbl=williamtable02 newtbl=williamtable02
INFO 05-07 16:11:32,706 - ugi=allwefantasy ip=unknown-ip-addr cmd=alter_table: db=default tbl=williamtable02 newtbl=williamtable02
INFO 05-07 16:11:32,939 - Updating table stats fast for williamtable02
INFO 05-07 16:11:32,939 - Updated size of table williamtable02 to 389088954
INFO 05-07 16:11:32,984 - Table default.williamtable02 stats: [numFiles=1, totalSize=389088954]
INFO 05-07 16:11:32,984 - </PERFLOG method=runTasks start=1467706291587 end=1467706292984 duration=1397 from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:32,984 - </PERFLOG method=Driver.execute start=1467706291580 end=1467706292984 duration=1404 from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:32,985 - OK
INFO 05-07 16:11:32,985 - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:32,985 - </PERFLOG method=releaseLocks start=1467706292985 end=1467706292985 duration=0 from=org.apache.hadoop.hive.ql.Driver>
INFO 05-07 16:11:32,985 - </PERFLOG method=Driver.run start=1467706290379 end=1467706292985 duration=2606 from=org.apache.hadoop.hive.ql.Driver>
res4: org.apache.spark.sql.SchemaRDD = []
scala> cc.sql("select count(*) from williamtable02").show
INFO 05-07 16:12:10,533 - main Query [SELECT COUNT(*) FROM WILLIAMTABLE02]
INFO 05-07 16:12:10,541 - Parsing command: select count(*) from williamtable02
INFO 05-07 16:12:10,553 - Parse Completed
INFO 05-07 16:12:10,569 - Parsing command: select count(*) from williamtable02
INFO 05-07 16:12:10,570 - Parse Completed
INFO 05-07 16:12:10,619 - 0: get_table : db=default tbl=williamtable02
INFO 05-07 16:12:10,619 - ugi=allwefantasy ip=unknown-ip-addr cmd=get_table : db=default tbl=williamtable02
INFO 05-07 16:12:10,731 - 0: get_table : db=default tbl=williamtable02
INFO 05-07 16:12:10,731 - ugi=allwefantasy ip=unknown-ip-addr cmd=get_table : db=default tbl=williamtable02
AUDIT 05-07 16:12:10,889 - [allwefantasy][allwefantasy][Thread-1]Table Not Found: williamtable02
org.spark-project.guava.util.concurrent.UncheckedExecutionException: org.apache.spark.sql.catalyst.analysis.NoSuchTableException
at org.spark-project.guava.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4882)
at org.spark-project.guava.cache.LocalCache$LocalLoadingCache.apply(LocalCache.java:4898)
at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:394)
at org.apache.spark.sql.CarbonContext$$anon$1.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(CarbonContext.scala:46)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:161)
at org.apache.spark.sql.CarbonContext$$anon$1.lookupRelation(CarbonContext.scala:46)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:303)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$9.applyOrElse(Analyzer.scala:315)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$9.applyOrElse(Analyzer.scala:310)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:56)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:54)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:54)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:281)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:321)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:54)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:310)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:300)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:83)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:80)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:80)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:72)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:72)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:36)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:36)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
at org.carbondata.spark.rdd.CarbonDataFrameRDD.<init>(CarbonDataFrameRDD.scala:23)
at org.apache.spark.sql.CarbonContext.sql(CarbonContext.scala:75)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:38)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:44)
at $iwC$$iwC$$iwC.<init>(<console>:46)
at $iwC$$iwC.<init>(<console>:48)
at $iwC.<init>(<console>:50)
at <init>(<console>:52)
at .<init>(<console>:56)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.sql.catalyst.analysis.NoSuchTableException
at org.apache.spark.sql.hive.CarbonMetastoreCatalog.lookupRelation1(CarbonMetastoreCatalog.scala:143)
at org.apache.spark.sql.CarbonDatasourceRelation.carbonRelation(CarbonDatasourceRelation.scala:137)
at org.apache.spark.sql.CarbonDatasourceRelation.<init>(CarbonDatasourceRelation.scala:146)
at org.apache.spark.sql.CarbonSource.createRelation(CarbonDatasourceRelation.scala:62)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
at org.apache.spark.sql.hive.HiveMetastoreCatalog$$anon$1.load(HiveMetastoreCatalog.scala:180)
at org.apache.spark.sql.hive.HiveMetastoreCatalog$$anon$1.load(HiveMetastoreCatalog.scala:124)
at org.spark-project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at org.spark-project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at org.spark-project.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at org.spark-project.guava.cache.LocalCache$Segment.get(LocalCache.java:2257)
at org.spark-project.guava.cache.LocalCache.get(LocalCache.java:4000)
at org.spark-project.guava.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at org.spark-project.guava.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at org.spark-project.guava.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4880)
... 91 more
scala>