You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/09/15 08:47:54 UTC
[GitHub] [iceberg] can-sun opened a new issue, #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
can-sun opened a new issue, #5768:
URL: https://github.com/apache/iceberg/issues/5768
### Apache Iceberg version
0.13.0
### Query engine
Spark
### Please describe the bug 🐞
iceberg-spark3-runtime version: 0.13.0
I attempted to skip the glut table name validation by setting **glue.skip-name-validation** to true. None of the following spark sql was successful.
iceberg catalog properties:
```
spark-shell --packages $DEPENDENCIES \
--conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.my_catalog.glue.skip-name-validation=true \
--conf spark.sql.catalog.my_catalog.warehouse=<s3-placeholder>\
--conf spark.sql.catalog.my_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
--conf spark.sql.catalog.my_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
```
spark sql queries tried so far
```
spark.sql("""CREATE TABLE IF NOT EXISTS my_catalog.db.`iceberg-table` ( id string,
creation_date string,
last_update_time string)
LOCATION '<my-s3-bucket>'
TBLPROPERTIES ('table_type'='ICEBERG', 'format'='parquet', 'glue.skip-name-validation'=true) """)
spark.sql("""CREATE TABLE IF NOT EXISTS my_catalog.db.`iceberg-table` (id string,
creation_date string,
last_update_time string)
USING iceberg
OPTIONS ( 'glue.skip-name-validation'=true )
LOCATION '<my-s3-bucket>'' """)
```
Error Stack trace:
```
java.lang.IllegalArgumentException: Invalid table identifier: db.iceberg-table
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:217)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.<init>(BaseMetastoreCatalog.java:115)
at org.apache.iceberg.BaseMetastoreCatalog.buildTable(BaseMetastoreCatalog.java:68)
at org.apache.iceberg.spark.SparkCatalog.newBuilder(SparkCatalog.java:578)
at org.apache.iceberg.spark.SparkCatalog.createTable(SparkCatalog.java:148)
at org.apache.iceberg.spark.SparkCatalog.createTable(SparkCatalog.java:92)
```
Besides I also tried to write data to the glue table, also failed. The ICEBERG table cannot be created via spark, however I can create such a table by using Athena query.
```
df.writeTo("my_catalog.db.`iceberg-table`").append()
```
Got table nor found
```
org.apache.spark.sql.AnalysisException: Table or view not found: my_catalog.db.`iceberg-table`;
'AppendData 'UnresolvedRelation [my_catalog, db, iceberg-table], [], false, true
+- Project [_1#3 AS id#10, _2#4 AS creation_date#11, _3#5 AS last_update_time#12]
+- LocalRelation [_1#3, _2#4, _3#5]
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:134)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:302)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:94)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis$(CheckAnalysis.scala:91)
at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:172)
at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:195)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330)
at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:192)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:90)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:192)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:224)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:224)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:90)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:88)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:95)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:136)
at org.apache.spark.sql.DataFrameWriterV2.runCommand(DataFrameWriterV2.scala:194)
at org.apache.spark.sql.DataFrameWriterV2.append(DataFrameWriterV2.scala:148)
```
I know I am not following the Glue/Athena best practices here: https://docs.aws.amazon.com/athena/latest/ug/glue-best-practices.html, however for purpose of backwards compatibility, I am still trying to figure out if it is viable to use dashes in ICEBERG table name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on issue #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on issue #5768:
URL: https://github.com/apache/iceberg/issues/5768#issuecomment-1248485869
> Does this mean I have to use a specific version of the iceberg-runtime jar according to the version of scala and spark?
Yup would have to use spark & scala version according to what the platform provides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] can-sun commented on issue #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
Posted by GitBox <gi...@apache.org>.
can-sun commented on issue #5768:
URL: https://github.com/apache/iceberg/issues/5768#issuecomment-1248482401
Gotcha, you are right, I just noticed that for >= 0.14.0 the jar was released based on different version of spark. Does this mean I have to use a specific version of the iceberg-runtime jar according to the version of scala and spark? Just wondered if there would be any incompatibility issues.
Let me try 0.14.0 first and see if I can make it : ) Thanks for your help, will update this open issue if I have any progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on issue #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on issue #5768:
URL: https://github.com/apache/iceberg/issues/5768#issuecomment-1248475013
Though I think documentation is a bit outdated ...
0.14.0 & 0.14.1 spark runtime jars should be of pattern : `iceberg-spark-runtime-{spark-version}_{scala_version}-{iceberg-version}`
for ex : iceberg version 0.14.1 with spark 3.3 and scala 2.1.2 should be
`iceberg-spark-runtime-3.3_2.1.2-0.14.1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on issue #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on issue #5768:
URL: https://github.com/apache/iceberg/issues/5768#issuecomment-1248155475
`glue.skip-name-validation` was introduced via https://github.com/apache/iceberg/commit/5653a0d9c704b22b3b193ae6338af6261833f6e2 which would be available iceberg version >= 0.14.0
based on above i think it's expected not be supported on 0.13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] can-sun closed issue #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
Posted by GitBox <gi...@apache.org>.
can-sun closed issue #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
URL: https://github.com/apache/iceberg/issues/5768
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] can-sun commented on issue #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
Posted by GitBox <gi...@apache.org>.
can-sun commented on issue #5768:
URL: https://github.com/apache/iceberg/issues/5768#issuecomment-1248621499
Thanks a lot! I just tried 0.14.1 and it works. Glue table can be created successfully and data insert also succeeded! I am closing this issue : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] can-sun commented on issue #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
Posted by GitBox <gi...@apache.org>.
can-sun commented on issue #5768:
URL: https://github.com/apache/iceberg/issues/5768#issuecomment-1248455341
Thanks! I saw the latest version released for spark3-runtime is **0.13.2**. Any plan for the release of >= 0.14.0?
```
# add Iceberg dependency
ICEBERG_VERSION=0.14.0
DEPENDENCIES="org.apache.iceberg:iceberg-spark3-runtime:$ICEBERG_VERSION"
# add AWS dependnecy
AWS_SDK_VERSION=2.17.131
AWS_MAVEN_GROUP=software.amazon.awssdk
AWS_PACKAGES=(
"bundle"
"url-connection-client"
)
for pkg in "${AWS_PACKAGES[@]}"; do
DEPENDENCIES+=",$AWS_MAVEN_GROUP:$pkg:$AWS_SDK_VERSION"
done
# start Spark SQL client shell
spark-sql --packages $DEPENDENCIES \
--conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \
--conf spark.sql.catalog.my_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
--conf spark.sql.catalog.my_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
```
The above code snippet is from documentation https://iceberg.apache.org/docs/latest/aws/
spark3-runtime 0.14.0 is used however I believed it was somehow to released yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on issue #5768: Spark cannot skip glue table name validation when creating ICEBERG tabel
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on issue #5768:
URL: https://github.com/apache/iceberg/issues/5768#issuecomment-1248465252
iceberg 0.14.0 and 0.14.1 both are already released :) !!
https://iceberg.apache.org/releases/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org