You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/31 05:50:38 UTC
[GitHub] [iceberg] zeromsi opened a new issue #3833: Failed to create iceberg table
zeromsi opened a new issue #3833:
URL: https://github.com/apache/iceberg/issues/3833
My code
```
SparkConf conf = new SparkConf();
conf.set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions");
conf.set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog");
conf.set("spark.sql.catalog.spark_catalog.type", "hive");
conf.set("spark.hadoop.hive.metastore.uris", "thrift://localhost:9083");
conf.set("spark.hadoop.hive.metastore.warehouse.dir", "s3a://resource-fvlelydg/warehouse/");
conf.set("spark.hadoop.fs.s3a.connection.ssl.enabled","true");
conf.set("spark.hadoop.fs.s3a.path.style.access","true");
conf.set("spark.hadoop.fs.s3a.endpoint","s3 endpoint");
conf.set("spark.hadoop.fs.s3a.access.key","access_key");
conf.set("spark.hadoop.fs.s3a.secret.key","secret_key");
conf.set("spark.hadoop.fs.s3a.attempts.maximum","1");
conf.set("spark.hadoop.fs.s3a.connection.establish.timeout","500");
conf.set("spark.hadoop.datanucleus.autoCreateSchema","true");
conf.set("spark.hadoop.datanucleus.fixedDatastore","false");
Schema schema = new Schema(
Types.NestedField.required(1, "level", Types.StringType.get()),
Types.NestedField.required(2, "event_time", Types.TimestampType.withZone()),
Types.NestedField.required(3, "message", Types.StringType.get()),
Types.NestedField.optional(4, "call_stack", Types.ListType.ofRequired(5, Types.StringType.get()))
);
SparkSession session = SparkSession.builder().appName("spark-test").master("local[3]")
.config(conf)
.getOrCreate();
session.catalog().createTable("test2", "/klovercloud");
```
### Error:
```
Exception in thread "main" java.lang.NoSuchMethodError: 'void scala.Function1.$init$(scala.Function1)'
at org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions.<init>(IcebergSparkSessionExtensions.scala:35)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347)
at java.base/java.lang.Class.newInstance(Class.java:645)
at org.apache.spark.sql.SparkSession$Builder.liftedTree1$1(SparkSession.scala:940)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:938)
at com.msi.spark.Maiin.main(Maiin.java:51)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on issue #3833: Failed to create table
Posted by GitBox <gi...@apache.org>.
flyrain commented on issue #3833:
URL: https://github.com/apache/iceberg/issues/3833#issuecomment-1006189591
You probably miss some scala lib in your job classpath. Here are a list of scala libs in my Spark job classpath for your reference.
```
scala-collection-compat_2.12-2.1.1.jar
scala-compiler-2.12.15.jar
scala-library-2.12.15.jar
scala-parser-combinators_2.12-1.1.2.jar
scala-reflect-2.12.15.jar
scala-xml_2.12-1.2.0.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zeromsi commented on issue #3833: Failed to create table
Posted by GitBox <gi...@apache.org>.
zeromsi commented on issue #3833:
URL: https://github.com/apache/iceberg/issues/3833#issuecomment-1003852016
Do I need to install Scala. I'm using Java here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #3833: Failed to create table
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3833:
URL: https://github.com/apache/iceberg/issues/3833#issuecomment-1003554291
looks like something is wrong with your Scala installation. Could you report your Spark environment and Scala version used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org