[GitHub] [iceberg] shiyuhang0 commented on issue #1756: iceberg is not a valid Spark SQL Data Source

[GitBox <gi...@apache.org>]

shiyuhang0 commented on issue #1756:
URL: https://github.com/apache/iceberg/issues/1756#issuecomment-1055665211


   > This occurs because of a bad error message in Spark. What it really means is that you attempted to read the table using a datasource v1 read path when the datasource in question is a v2 one. This is occuring because you are using the native spark session catalog rather than the iceberg v2 replacement session catalog. So from here you have a two options I think, You can override the session catalog with the iceberg V2 session catalog You can read the table using the v2 path explicitly, spark.read.table()
   > […](#)
   > On Thu, Nov 12, 2020, 5:47 AM chaiyuan2046 ***@***.***> wrote: spark : spark-3.0.1-bin-hadoop2.7 iceberg : iceberg-spark3-runtime-0.9.1.jar ============================================== *Step one:* bin/spark-sql --conf spark.sql.warehouse.dir=hdfs://xxxx:8020/user/iceberg *Step two:* CREATE TABLE ib_test(id bigint, data string) USING iceberg; --success *Step three:* select * from ib_test; --The error blow happens： Caused by: org.apache.spark.sql.AnalysisException: iceberg is not a valid Spark SQL Data Source.; at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:421) at org.apache.spark.sql.execution.datasources.FindDataSourceTable.$anonfun$readDataSourceTable$1(DataSourceStrategy.scala:256) at org.sparkproject.guava.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792) at org.sparkproject.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599) at org.sparkproject.guava.cache.LocalCache$Segment.loadSync(Local
 Cache.java:2379) ... 96 more Any ideas for this question？tks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1756>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADE2YPBNYRXGD3YDBWJ5PDSPPDM7ANCNFSM4TTFYMWQ> .
   
   
   I am new to use spark and I would like to know how to  `override the session catalog with the iceberg V2 session catalog`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org