You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/06 08:58:10 UTC
[GitHub] [iceberg] itaise opened a new issue, #4510: Spark overwrite issue when partition changes
itaise opened a new issue, #4510:
URL: https://github.com/apache/iceberg/issues/4510
Hi,
we are writing to iceberg using spark, and when renaming the partition field name, we are getting a validation error:
```
org.apache.iceberg.exceptions.ValidationException: Cannot find source column for partition field: 1000: some_date: void(1)
```
It seems like iceberg is referring to the existing table partition field name, which is irrelevant anymore - as there is a new partition field, and the write mode is "overwrite".
Can you please assist?
Thank you!
Here is a minimal reproducible example:
1. create the original table with partition field "some_date":
```
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType ,StructField, StringType
dataDF = [('1991-04-01',)]
schema = StructType([
StructField('some_date',StringType(), True)])
spark = SparkSession.builder.master('local[1]').appName('example') \
.getOrCreate()
df = spark.createDataFrame(data = dataDF, schema = schema)
spark.sql(f"use iprod") # catalog
spark.sql(f"CREATE SCHEMA IF NOT EXISTS iprod.test_schema")
df.write.mode("overwrite").format("parquet").partitionBy('some_date').saveAsTable("iprod.test_schema.example")
```
2. Try to overwrite the table with the same code, but the partition field renamed to ```some_date_2```
```
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType ,StructField, StringType
dataDF = [('1991-04-01',)]
schema = StructType([
StructField('some_date_2',StringType(), True)])
spark = SparkSession.builder.master('local[1]').appName('example') \
.getOrCreate()
df = spark.createDataFrame(data = dataDF, schema = schema)
spark.sql(f"use iprod") # catalog
spark.sql(f"CREATE SCHEMA IF NOT EXISTS iprod.test_schema")
df.write.mode("overwrite").format("parquet").partitionBy('some_date_2').saveAsTable("iprod.test_schema.example")
```
Full trace:
```
: org.apache.iceberg.exceptions.ValidationException: Cannot find source column for partition field: 1000: some_date: void(1)
at org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:46)
at org.apache.iceberg.PartitionSpec.checkCompatibility(PartitionSpec.java:511)
at org.apache.iceberg.PartitionSpec$Builder.build(PartitionSpec.java:503)
at org.apache.iceberg.TableMetadata.reassignPartitionIds(TableMetadata.java:768)
at org.apache.iceberg.TableMetadata.buildReplacement(TableMetadata.java:790)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.newReplaceTableTransaction(BaseMetastoreCatalog.java:256)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.createOrReplaceTransaction(BaseMetastoreCatalog.java:244)
at org.apache.iceberg.CachingCatalog$CachingTableBuilder.createOrReplaceTransaction(CachingCatalog.java:244)
at org.apache.iceberg.spark.SparkCatalog.stageCreateOrReplace(SparkCatalog.java:190)
at org.apache.spark.sql.execution.datasources.v2.AtomicReplaceTableAsSelectExec.run(WriteToDataSourceV2Exec.scala:197)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.doExecute(V2CommandExec.scala:55)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:194)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:232)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:229)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:190)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:134)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:133)
at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:686)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:619)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:750)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] itaise commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
itaise commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1163062143
@romibuzi thanks i will try it out and update!
@rdblue thanks a lot! What do you mean by updating the table to V2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark overwrite issue when partition changes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-2303330477
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1163236562
@itaise, by default tables use the v1 format, which didn't use unique IDs for partition fields in metadata. That was fixed in v2. So when you upgrade a table to v2, (set the `table-format` property to 2) your changes will actually remove the unnecessary field that references the old column, rather than keeping a placeholder partition field around for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1281115035
@Fokko, can you check if this is still a problem after your recent fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rajarshisarkar commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
rajarshisarkar commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1090049772
I can't find the table being altered to introduce the new partition field `some_date_2`. Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] itaise commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
itaise commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1101380297
This happens also when using the DataFrameWriter V2, for example:
`df.writeTo("iprod.test_schema.example").partitionedBy(partition_column).createOrReplace()`
and also with:
`df.writeTo("iprod.test_schema.example").partitionedBy(partition_column).replace()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] itaise commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
itaise commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1101354085
@rdblue We will highly appreciate your input on that, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1161912109
I think the problem is that this is a v1 table. In v1, the `some_date` partition field is not actually dropped, it is replaced with a null / void transform. So you still need the original column ref. If you update your table to v2 and then try the write it should work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] itaise commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
itaise commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1318450967
Hi :) any update regarding that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] itaise commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
itaise commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1090067882
@rajarshisarkar the first part is creating the first version, and the second part tries to overwrite it.
Maybe you should change the catalog name to your catalog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] romibuzi commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
romibuzi commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1161321533
Hi @itaise,
you need to change the partition specs of your table before writing the data with the new partition:
```
spark.sql("ALTER TABLE my_iceberg_catalog.db.vaccinations DROP PARTITION FIELD some_date")
spark.sql("ALTER TABLE my_iceberg_catalog.db.vaccinations ADD PARTITION FIELD some_date_2")
```
This will not drop `some_date` from your data but only metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark overwrite issue when partition changes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4510: Spark overwrite issue when partition changes
URL: https://github.com/apache/iceberg/issues/4510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] itaise commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
itaise commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1173428712
thank you so much @rdblue!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] itaise commented on issue #4510: Spark overwrite issue when partition changes
Posted by GitBox <gi...@apache.org>.
itaise commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-1177506225
Thank you @rdblue - I tried your suggestion but unfortunately getting another error, I added the following just before the table overwrite and it didn't help:
`spark.sql(f"""ALTER TABLE iprod.test_schema.example SET TBLPROPERTIES ('format-version' = '2')""")`
as mentioned in the Tabular article: https://tabular.io/blog/iceberg-format-version/
and getting this error:
`Caused by: java.lang.NullPointerException: Cannot find source column: 1
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:953)
at org.apache.iceberg.PartitionSpec$Builder.add(PartitionSpec.java:490)
at org.apache.iceberg.PartitionSpecParser.buildFromJsonFields(PartitionSpecParser.java:155)
at org.apache.iceberg.PartitionSpecParser.fromJson(PartitionSpecParser.java:78)
at org.apache.iceberg.PartitionSpecParser.lambda$fromJson$0(PartitionSpecParser.java:91)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2344)
at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2342)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2325)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
at org.apache.iceberg.PartitionSpecParser.fromJson(PartitionSpecParser.java:88)
at org.apache.iceberg.SerializableTable.lambda$specs$2(SerializableTable.java:186)
at java.util.HashMap.forEach(HashMap.java:1290)
at org.apache.iceberg.SerializableTable.specs(SerializableTable.java:185)
at org.apache.iceberg.SerializableTable.spec(SerializableTable.java:175)
at org.apache.iceberg.io.OutputFileFactory$Builder.<init>(OutputFileFactory.java:152)
at org.apache.iceberg.io.OutputFileFactory$Builder.<init>(OutputFileFactory.java:140)
at org.apache.iceberg.io.OutputFileFactory.builderFor(OutputFileFactory.java:108)
at org.apache.iceberg.spark.source.SparkWrite$WriterFactory.createWriter(SparkWrite.java:542)
at org.apache.iceberg.spark.source.SparkWrite$WriterFactory.createWriter(SparkWrite.java:535)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:408)
at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:360)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark overwrite issue when partition changes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4510:
URL: https://github.com/apache/iceberg/issues/4510#issuecomment-2274584159
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org