You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/05/31 11:57:25 UTC
[GitHub] [iceberg] ashoksri0 opened a new issue #2652: Spark sql Create DDL not wokring
ashoksri0 opened a new issue #2652:
URL: https://github.com/apache/iceberg/issues/2652
**spark version: 3.1.1**
**iceberg version: 0.11.1**
**Source Code:**
`
object IcebergIngestionDDLTest {
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder()
.master("spark://ASH:7077")
.appName("SparkByExample")
.appName("Spark Hive Example")
.config("packages","org.apache.iceberg:iceberg-spark3-runtime:0.11.1")
.config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.config("spark.sql.catalog.spark_catalog","org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type","hive")
.config("spark.sql.catalog.local","org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.local.type","hadoop")
.config("spark.sql.catalog.local.warehouse","hdfs://localhost:54310/lwa/data/")
.config("spark.sql.warehouse.dir","hdfs://localhost:54310/lwa/data/")
.config("iceberg.engine.hive.enabled", "true")
.config("spark.sql.catalog.hive_prod.uri","thrift://127.0.0.1:10000")
.enableHiveSupport()
.getOrCreate()
spark.sql("CREATE TABLE local.default.sample ( id bigint COMMENT 'unique id', data string) USING iceberg");
}
}
`
**Logs:**
`Exception in thread "main" java.lang.NoSuchMethodError: 'void org.apache.spark.sql.internal.VariableSubstitution.<init>(org.apache.spark.sql.internal.SQLConf)'
at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.substitutor$lzycompute(IcebergSparkSqlExtensionsParser.scala:39)
at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.substitutor(IcebergSparkSqlExtensionsParser.scala:39)
at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parsePlan(IcebergSparkSqlExtensionsParser.scala:96)
at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:613)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:613)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
at com.p3.ingestion.demo.IcebergIngestionDDLTest$.main(IcebergIngestionDDLTest.scala:28)
at com.p3.ingestion.demo.IcebergIngestionDDLTest.main(IcebergIngestionDDLTest.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ashoksri0 closed issue #2652: Spark sql Create DDL not wokring
Posted by GitBox <gi...@apache.org>.
ashoksri0 closed issue #2652:
URL: https://github.com/apache/iceberg/issues/2652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on issue #2652: Spark sql Create DDL not wokring
Posted by GitBox <gi...@apache.org>.
rymurr commented on issue #2652:
URL: https://github.com/apache/iceberg/issues/2652#issuecomment-851444836
Hi @ashoksri0
Can you try with Spark `3.0.2`? The `0.11.x` release isn't compatible with Spark `3.1.x`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ashoksri0 commented on issue #2652: Spark sql Create DDL not wokring
Posted by GitBox <gi...@apache.org>.
ashoksri0 commented on issue #2652:
URL: https://github.com/apache/iceberg/issues/2652#issuecomment-851893859
@rymurr thanks, now its working with 3.0.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org