You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Zhanxiao-Ma (via GitHub)" <gi...@apache.org> on 2024/03/28 02:23:41 UTC
[I] Rewrite Files Action OOM [iceberg]
Zhanxiao-Ma opened a new issue, #10054:
URL: https://github.com/apache/iceberg/issues/10054
### Query engine
Spark
### Question
V2 table support equality deletes for row level delete. When I use the Java API to write a large number of delete records and then start the spark rewrite files Action, I get an OOM error. Is it not allowed to delete too many records? How can I resolve this issue?
Does the community have plans to improve this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "nk1506 (via GitHub)" <gi...@apache.org>.
nk1506 commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2041538414
> @nk1506 Echoing Russell's comments, how many small files are there in your OOM case? How much memory do you set up?
I didn't use spark-engine for compaction. I was using Java Client API. My queries might distract from the original problem. Although my requirement is to compact very large datasets(say 10K datafiles) with single commit. Using [RewriteFiles](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/RewriteFiles.java#L171) always might cause OOM. So I am looking something which can help to manage manifestFiles more intelligently. I think I will start different thread to discuss the other problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "Zhanxiao-Ma (via GitHub)" <gi...@apache.org>.
Zhanxiao-Ma commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2026552267
> There really isn't enough information here to dig into the issue. How many records are there, what were the spark settings, did it not oom before there were deletes? Did it oom during shuffle? Did the executors OOM? Was an unreasonable amount of memory being consumed?
>
> As a general statement we are interested in improving performance but OOM's can happen for many different reasons so it's not something that can be universally fixed.
OK. Actually, this line of code is causing the OOM error because it loads all EqDelete record into memory. And as the number of records to be deleted increases, the memory requirement also increases.
https://github.com/apache/iceberg/blob/2d76c91d6a27378fe753eb7f912bf3031368e6b0/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L190
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "pdames (via GitHub)" <gi...@apache.org>.
pdames commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2121567721
Any updates here @Zhanxiao-Ma? Would love to take a look at what you've implemented if you've got a pending PR to link back to this issue, and see if there's an opportunity to work together to improve the state of affairs here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2026137455
There really isn't enough information here to dig into the issue. How many records are there, what were the spark settings, did it not oom before there were deletes? Did it oom during shuffle? Did the executors OOM? Was an unreasonable amount of memory being consumed?
As a general statement we are interested in improving performance but OOM's can happen for many different reasons so it's not something that can be universally fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2571807122
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2218139056
I've created a [draft PR](https://github.com/apache/iceberg/pull/10667) which has been verified in our environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10054: Spark rewrite Files Action OOM
URL: https://github.com/apache/iceberg/issues/10054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2603392960
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2024928341
It's not forbidden to delete too many records but could increase memory required in the driver. If you are using position deletes, there's `rewrite_position_delete_files`. As for equality deletes, there was https://github.com/apache/iceberg/pull/2364 to rewrite equality deletes as position deletes but not merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "Zhanxiao-Ma (via GitHub)" <gi...@apache.org>.
Zhanxiao-Ma commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2041085307
> @nk1506 Echoing Russell's comments, how many small files are there in your OOM case? How much memory do you set up?
@RussellSpitzer I believe increasing memory is not a good solution for dealing with excessive information deletion because it is impossible to predict how much memory would be appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "nk1506 (via GitHub)" <gi...@apache.org>.
nk1506 commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2039248808
@RussellSpitzer / @manuzhang , are we planning to make any fix for this? OOM has been observed with [RewriteFiles](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/RewriteFiles.java#L171) too?
If we use this API with large chunks of small files to be rewritten with new large files, It causes OOM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2100175845
> I have implemented a disk-based map to solve this problem. Is this what Iceberg expects? If so, I will submit the code.
@Zhanxiao-Ma I think it will be valuable to the community. Please open a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "Zhanxiao-Ma (via GitHub)" <gi...@apache.org>.
Zhanxiao-Ma commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2041085342
@RussellSpitzer I have implemented a disk-based map to solve this problem. Is this what Iceberg expects? If so, I will submit the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark rewrite Files Action OOM [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2039934176
@nk1506 Echoing Russell's comments, how many small files are there in your OOM case? How much memory do you set up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org