You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "AllenWee1106 (via GitHub)" <gi...@apache.org> on 2023/11/29 07:38:05 UTC
[I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
AllenWee1106 opened a new issue, #9178:
URL: https://github.com/apache/iceberg/issues/9178
### Query engine
spark
### Question
I submitted a task through Spark and it stopped due to an exception. The reason for the exception is as follows:
`Caused by: org.apache.iceberg.exceptions.CommitFailedException: Cannot commit datalake.slp_quality: metadata location s3://warehouse/datalake/SLP_Quality/metadata/6390c7416eff.metadata.json has changed from s3://warehouse/datalake/SLP_Quality/metadata/b167af31b5fa.metadata.json
at org.apache.iceberg.jdbc.JdbcTableOperations.validateMetadataLocation(JdbcTableOperations.java:211)
at org.apache.iceberg.jdbc.JdbcTableOperations.doCommit(JdbcTableOperations.java:111)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:389)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:363)
at org.apache.iceberg.spark.source.SparkWrite.commitOperation(SparkWrite.java:206)
at org.apache.iceberg.spark.source.SparkWrite.access$1300(SparkWrite.java:82)
at org.apache.iceberg.spark.source.SparkWrite$BatchAppend.commit(SparkWrite.java:269)
at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:369)`
I would like to know, what is the reason for this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-2625997929
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9178: An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed
URL: https://github.com/apache/iceberg/issues/9178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "Zhangg7723 (via GitHub)" <gi...@apache.org>.
Zhangg7723 commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1860081404
> @Zhangg7723  Thank you. Is the method set in the above picture ? Or use table. properties. put ()?
use set() and don't forget to commit this update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "Zhangg7723 (via GitHub)" <gi...@apache.org>.
Zhangg7723 commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1857382064
From https://iceberg.apache.org/docs/1.2.1/configuration/#write-properties
write.update.isolation-level, write.merge.isolation-level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "AllenWee1106 (via GitHub)" <gi...@apache.org>.
AllenWee1106 commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1855434363
@Zhangg7723 thx, how to change the isolation level to snapshot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "nk1506 (via GitHub)" <gi...@apache.org>.
nk1506 commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1833467461
As per implementation [validateMetadataLocation](https://github.com/apache/iceberg/blob/d247b20f166ccb0b92443d4b05330b1e0d9c5d49/core/src/main/java/org/apache/iceberg/jdbc/JdbcTableOperations.java#L111) it is validating old metadataLocation before commits in case of any writes. I am making a guess the heavy concurrent operation can cause this issue. Tagging @ismailsimsek , if he can share his insights.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "Zhangg7723 (via GitHub)" <gi...@apache.org>.
Zhangg7723 commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1840183519
maybe you can change the isolation level to snapshot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "AllenWee1106 (via GitHub)" <gi...@apache.org>.
AllenWee1106 commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1832991874
@nk1506
Thank you for your reply.
Based on your analysis, I have the following questions
1. I have multiple Spark tasks running at the same time. According to your analysis, is it possible that among these tasks, data was written to the same Iceberg table at the same time, which led to the occurrence of this problem?
2. In coding, I used a loop approach to execute Spark code (but the loop also executes one after another, not in parallel). Is this the reason for the problem?
=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "AllenWee1106 (via GitHub)" <gi...@apache.org>.
AllenWee1106 commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1835398678
@nk1506 @ismailsimsek THX, What is the maximum concurrent write supported by Iceberg?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "ismailsimsek (via GitHub)" <gi...@apache.org>.
ismailsimsek commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1860252682
@AllenWee1106 could you also try with higher `commit.retry.num-retries` (default is 4) . to see if this decreases the conflicts. this is another [table property ](https://iceberg.apache.org/spec/#table-metadata-fields). or check it in debug logs to see if commit-retry happening at all?
if that doesn't help, could try to add retry manually where you do commit operation. (Im not sure if the catalogs do retry operation by default)
[more detail on commit resolution](https://iceberg.apache.org/spec/#commit-conflict-resolution-and-retry)
an example:
https://github.com/apache/iceberg/blob/9342f64a713ed4a1c540046ae5db91b3ac86ac02/core/src/main/java/org/apache/iceberg/BaseReplaceSortOrder.java#L39-L67
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "srcnblgc (via GitHub)" <gi...@apache.org>.
srcnblgc commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-2094374588
Possible to let me know how I can increase the number of retries on glue backed environment?
Or how to set "From https://iceberg.apache.org/docs/1.2.1/configuration/#write-properties
write.update.isolation-level, write.merge.isolation-level" via python(which will be running on lambda) with glue backed environment?
here what I was trying:
------------------------------------------------------------------
catalog = load_catalog("glue", **{"type": "glue"})
df = pq.read_table("~/Desktop/pyiceberg_py/yellow_tripdata_2023-01.parquet")
tbl = catalog.load_table(identifier="pyiceberg_sb.pyiceberg_nyc_taxi")
tbl.append(df)
print(len(tbl.scan().to_arrow()))
------------------------------------------------------------------
and it is failing with "pyiceberg.exceptions.CommitFailedException: Requirement failed: branch main has changed: expected id 3182993835964802089, found 1838899954983424460" with only 2 concurrent inserts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "nk1506 (via GitHub)" <gi...@apache.org>.
nk1506 commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1831614906
it seems concurrent commit issues. Is there by an chance you are running these queries in parallel?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "ismailsimsek (via GitHub)" <gi...@apache.org>.
ismailsimsek commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1833596213
@nk1506 @AllenWee1106 correct that will be my guess to, concurrent write to same table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "AllenWee1106 (via GitHub)" <gi...@apache.org>.
AllenWee1106 commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1860003242
@Zhangg7723
Thank you. Is the method set in the above picture ? Or use table. properties. put ()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9178:
URL: https://github.com/apache/iceberg/issues/9178#issuecomment-2594192321
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org