You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/08/31 04:08:10 UTC

[GitHub] [iceberg] dmgcodevil opened a new issue, #5675: Limit the number of files for rewrite/compaction action

dmgcodevil opened a new issue, #5675:
URL: https://github.com/apache/iceberg/issues/5675

   ### Query engine
   
   Flink
   
   ### Question
   
   We have a streaming Flink job that continously consumes records from Kafka and stores them into Iceberg. 
   The [RewriteDataFilesAction](https://github.com/apache/iceberg/blob/master/flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java) is getting slower over time and `TaskManager` is starting to fails with OOM. Is it possible to limit the number of data files for compaction? I have the following ideas:
   
   1. Modify https://github.com/apache/iceberg/blob/master/flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java#L49 to limit a number of `FileScanTask` in `CombinedScanTask`. We can create some sort of _wrapper_:
   
   ```java
       class BoundedCombinedScanTask implements CombinedScanTask {
           private final CombinedScanTask original;
           private final int limit;
   
           public BoundedCombinedScanTask(CombinedScanTask original, int limit) {
               this.original = original;
               this.limit = limit;
           }
   
           @Override
           public Collection<FileScanTask> files() {
               return original.files().stream().limit(limit).collect(Collectors.toList());
           }
   
           @Override
           public CombinedScanTask asCombinedScanTask() {
               return original;
           }
       }
   ```
   
   Flink code will look as follows:
   
   ```java
   protected List<DataFile> rewriteDataForTasks(final List<CombinedScanTask> combinedScanTasks) {
           final List<CombinedScanTask boundedCombinedScanTasks = combinedScanTasks.stream(t -> new BoundedCombinedScanTask(t, 100)).collect(Collectors.toList());
           int size = boundedCombinedScanTasks .size();
           int parallelism = Math.min(size, this.maxParallelism);
           DataStream<CombinedScanTask> dataStream = this.env.fromCollection(boundedCombinedScanTasks );
           RowDataRewriter rowDataRewriter = new RowDataRewriter(this.table(), this.caseSensitive(), this.fileIO(), this.encryptionManager());
   
           try {
               return rowDataRewriter.rewriteDataForTasks(dataStream, parallelism);
           } catch (Exception var7) {
               throw new RuntimeException("Rewrite data file error.", var7);
           }
       }
   ```
   
   2. Modify [BaseRewriteDataFilesAction](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java) https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java#L268
   
   So that each group will contain at most N files.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] github-actions[bot] closed issue #5675: Limit the number of files for rewrite/compaction action

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5675: Limit the number of files for rewrite/compaction action
URL: https://github.com/apache/iceberg/issues/5675


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] nastra commented on issue #5675: Limit the number of files for rewrite/compaction action

Posted by GitBox <gi...@apache.org>.
nastra commented on issue #5675:
URL: https://github.com/apache/iceberg/issues/5675#issuecomment-1325108619

   @dmgcodevil did you try solving your issue by setting `read.split.planning-lookback`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] dmgcodevil commented on issue #5675: Limit the number of files for rewrite/compaction action

Posted by GitBox <gi...@apache.org>.
dmgcodevil commented on issue #5675:
URL: https://github.com/apache/iceberg/issues/5675#issuecomment-1325307067

   @nastra do you mean [this](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java#L167) ? What is supposed to do?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] stevenzwu commented on issue #5675: Limit the number of files for rewrite/compaction action

Posted by GitBox <gi...@apache.org>.
stevenzwu commented on issue #5675:
URL: https://github.com/apache/iceberg/issues/5675#issuecomment-1239858884

   @dmgcodevil not quite following option 1 that you described. that will skip some files for compaction, right? 
   
   seems like the `lookback` is the right knob for your problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] nastra commented on issue #5675: Limit the number of files for rewrite/compaction action

Posted by GitBox <gi...@apache.org>.
nastra commented on issue #5675:
URL: https://github.com/apache/iceberg/issues/5675#issuecomment-1325322119

   This is the `lookback` parameter that is being passed to `BinPacking.PackingIterable`. According to the docs it is `Number of bins to consider when combining input splits`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] dmgcodevil commented on issue #5675: Limit the number of files for rewrite/compaction action

Posted by GitBox <gi...@apache.org>.
dmgcodevil commented on issue #5675:
URL: https://github.com/apache/iceberg/issues/5675#issuecomment-1325333076

   @nastra oh, right, we tried that. It limits the number of bins while iterating, however, the whole input will be processed anyway.
   see https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/util/BinPacking.java#L122-L154


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] dmgcodevil commented on issue #5675: Limit the number of files for rewrite/compaction action

Posted by GitBox <gi...@apache.org>.
dmgcodevil commented on issue #5675:
URL: https://github.com/apache/iceberg/issues/5675#issuecomment-1311065885

   PR: https://github.com/apache/iceberg/pull/6174


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] github-actions[bot] commented on issue #5675: Limit the number of files for rewrite/compaction action

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5675:
URL: https://github.com/apache/iceberg/issues/5675#issuecomment-1613941844

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] github-actions[bot] commented on issue #5675: Limit the number of files for rewrite/compaction action

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5675:
URL: https://github.com/apache/iceberg/issues/5675#issuecomment-1635079083

   This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org