You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "BsoBird (via GitHub)" <gi...@apache.org> on 2024/12/14 02:47:54 UTC
[PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
BsoBird opened a new pull request, #11787:
URL: https://github.com/apache/iceberg/pull/11787
…ution
If data in the source table is modified during the execution of the merge-into command, it may lead to incorrect results for the MergeIntoCommand. We need to alert users to avoid this risk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #11787:
URL: https://github.com/apache/iceberg/pull/11787#discussion_r1886334243
##########
docs/docs/spark-writes.md:
##########
@@ -101,6 +101,9 @@ Spark 3.5 added support for `WHEN NOT MATCHED BY SOURCE ... THEN ...` to update
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET status = 'invalid'
```
+!!! danger
+ Note: For copy on write table,Please Do Not Modify the Source Data Table During Execution.Due to the need for Spark to use the source table consecutively twice for computation,the relation which is created of source data must remain constant through the two different passes of the source data.If the source data query would return different results user will see odd behavior.
Review Comment:
> Would it be possible to be more specific? What kind of odd behavior users will experience
The most common situation encountered in our production environment is data loss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #11787:
URL: https://github.com/apache/iceberg/pull/11787#discussion_r1886334243
##########
docs/docs/spark-writes.md:
##########
@@ -101,6 +101,9 @@ Spark 3.5 added support for `WHEN NOT MATCHED BY SOURCE ... THEN ...` to update
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET status = 'invalid'
```
+!!! danger
+ Note: For copy on write table,Please Do Not Modify the Source Data Table During Execution.Due to the need for Spark to use the source table consecutively twice for computation,the relation which is created of source data must remain constant through the two different passes of the source data.If the source data query would return different results user will see odd behavior.
Review Comment:
> Would it be possible to be more specific? What kind of odd behavior users will experience
Data loss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #11787:
URL: https://github.com/apache/iceberg/pull/11787#discussion_r1886334243
##########
docs/docs/spark-writes.md:
##########
@@ -101,6 +101,9 @@ Spark 3.5 added support for `WHEN NOT MATCHED BY SOURCE ... THEN ...` to update
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET status = 'invalid'
```
+!!! danger
+ Note: For copy on write table,Please Do Not Modify the Source Data Table During Execution.Due to the need for Spark to use the source table consecutively twice for computation,the relation which is created of source data must remain constant through the two different passes of the source data.If the source data query would return different results user will see odd behavior.
Review Comment:
> Would it be possible to be more specific? What kind of odd behavior users will experience
@Fokko Hello,Sir. The most common situation encountered in our production environment is data loss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #11787:
URL: https://github.com/apache/iceberg/pull/11787#issuecomment-2557394434
-1
I don't think it is appropriate to add danger warnings to Iceberg docs for bugs like this one and if we did I think this is not clear enough about the cause. It looks like the issue is that the source table ("For example:hive table,jdbc view table,another datalake table,etc.") does not provide repeatable reads for a query. Since this plan is in Spark, that's a better place to document the behavior.
In addition, I don't know that this is valuable. If you use unreliable source data (i.e. Hive tables), it seems like you should not have expectations of strict behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #11787:
URL: https://github.com/apache/iceberg/pull/11787#discussion_r1888049453
##########
docs/docs/spark-writes.md:
##########
@@ -101,6 +101,9 @@ Spark 3.5 added support for `WHEN NOT MATCHED BY SOURCE ... THEN ...` to update
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET status = 'invalid'
```
+!!! danger
+ Note: For copy on write table,Please Do Not Modify the Source Data Table During Execution.Due to the need for Spark to use the source table consecutively twice for computation,the relation which is created of source data must remain constant through the two different passes of the source data.If the source data query would return different results user will see odd behavior.
Review Comment:
I'm trying to pierce the parts together, from Slack I understand that the source table is a Hive table, not Iceberg. Is that correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #11787:
URL: https://github.com/apache/iceberg/pull/11787#discussion_r1886329539
##########
docs/docs/spark-writes.md:
##########
@@ -101,6 +101,9 @@ Spark 3.5 added support for `WHEN NOT MATCHED BY SOURCE ... THEN ...` to update
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET status = 'invalid'
```
+!!! danger
+ Note: For copy on write table,Please Do Not Modify the Source Data Table During Execution.Due to the need for Spark to use the source table consecutively twice for computation,the relation which is created of source data must remain constant through the two different passes of the source data.If the source data query would return different results user will see odd behavior.
Review Comment:
Would it be possible to be more specific? What kind of odd behavior users will experience
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird closed pull request #11787: Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec…
URL: https://github.com/apache/iceberg/pull/11787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on PR #11787:
URL: https://github.com/apache/iceberg/pull/11787#issuecomment-2543564046
@RussellSpitzer
Hello Sir,I have modified the SQL used in our production environment according to your suggestions, and currently, it appears that the issue of data loss has indeed disappeared. This confirms that your judgment was correct. Given that many users have not yet noticed this issue, I suggest that we need to explain this problem to users in the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #11787:
URL: https://github.com/apache/iceberg/pull/11787#discussion_r1888149882
##########
docs/docs/spark-writes.md:
##########
@@ -101,6 +101,9 @@ Spark 3.5 added support for `WHEN NOT MATCHED BY SOURCE ... THEN ...` to update
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET status = 'invalid'
```
+!!! danger
+ Note: For copy on write table,Please Do Not Modify the Source Data Table During Execution.Due to the need for Spark to use the source table consecutively twice for computation,the relation which is created of source data must remain constant through the two different passes of the source data.If the source data query would return different results user will see odd behavior.
Review Comment:
> I'm trying to pierce the parts together, from Slack I understand that the source table is a Hive table, not Iceberg. Is that correct?
@Fokko Yea,Sir. But in fact, other non-iceberg types of source tables could also cause this issue.
Therefore, it would be best for us to do this:
```
before:
merge into target_iceberg_cow_table t using (select * from dynamic_incr_table) s when xxxx; --data loss
after:
--for hive table(or external table)/jdbc view table,etc
create table static_table as select * from dynamic_incr_table; --write a temp table(data not change)
merge into target_iceberg_cow_table t using (select * from static_table ) s when xxxx;
--for another datalake table
merge into target_iceberg_cow_table t using (select * from dynamic_incr_table TIMESTAMP AS OF '2021-09-20 08:00:00') s when xxxx; --read a specific version
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on PR #11787:
URL: https://github.com/apache/iceberg/pull/11787#issuecomment-2558027178
@rdblue Sir, the execution plans for MOR tables and COW tables when executing MergeIntoCommand are different. For MOR tables, since the source table is only used once, such issues do not arise. Given that the factor determining Spark's execution plan is the type of Iceberg table used by the user, it seems inappropriate to document this issue on the Spark side as well. So, where should this issue be documented then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org