You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wypoon (via GitHub)" <gi...@apache.org> on 2023/10/12 01:24:45 UTC

[PR] Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

wypoon opened a new pull request, #8805:
URL: https://github.com/apache/iceberg/pull/8805

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on code in PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#discussion_r1372439543


##########
core/src/test/java/org/apache/iceberg/deletes/TestPositionFilter.java:
##########
@@ -282,6 +286,16 @@ public void testPositionSetRowFilter() {
 
   @Test
   public void testCombinedPositionSetRowFilter() {
+    testCombinedPositionSetRowFilter(null);

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#discussion_r1376017012


##########
core/src/test/java/org/apache/iceberg/deletes/TestPositionFilter.java:
##########
@@ -280,8 +287,16 @@ public void testPositionSetRowFilter() {
         .containsExactlyElementsOf(Lists.newArrayList(1L, 2L, 5L, 6L, 8L));
   }
 
-  @Test
-  public void testCombinedPositionSetRowFilter() {
+  static Stream<ExecutorService> executorServiceProvider() {
+    ExecutorService executorService =

Review Comment:
   nit: maybe this can be inlined?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#discussion_r1366551865


##########
core/src/test/java/org/apache/iceberg/deletes/TestPositionFilter.java:
##########
@@ -282,6 +286,16 @@ public void testPositionSetRowFilter() {
 
   @Test
   public void testCombinedPositionSetRowFilter() {
+    testCombinedPositionSetRowFilter(null);

Review Comment:
   minor: why not make the test a `@ParameterizedTest`, taking a `null` and a configured `ExecutorService`? I think that would better express that this is being tested with different parameters if makes it easier to debug when one of the parameters fail for example



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on code in PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#discussion_r1378364590


##########
core/src/main/java/org/apache/iceberg/deletes/Deletes.java:
##########
@@ -127,14 +130,25 @@ public static StructLikeSet toEqualitySet(
 
   public static <T extends StructLike> PositionDeleteIndex toPositionIndex(
       CharSequence dataLocation, List<CloseableIterable<T>> deleteFiles) {
+    return toPositionIndex(dataLocation, deleteFiles, ThreadPools.getDeleteWorkerPool());
+  }
+
+  public static <T extends StructLike> PositionDeleteIndex toPositionIndex(
+      CharSequence dataLocation,
+      List<CloseableIterable<T>> deleteFiles,
+      ExecutorService deleteWorkerPool) {
     DataFileFilter<T> locationFilter = new DataFileFilter<>(dataLocation);
     List<CloseableIterable<Long>> positions =
         Lists.transform(
             deleteFiles,
             deletes ->
                 CloseableIterable.transform(
                     locationFilter.filter(deletes), row -> (Long) POSITION_ACCESSOR.get(row)));
-    return toPositionIndex(CloseableIterable.concat(positions));
+    if (positions.size() > 1 && (deleteWorkerPool != null)) {

Review Comment:
   Done.



##########
core/src/test/java/org/apache/iceberg/deletes/TestPositionFilter.java:
##########
@@ -280,8 +287,16 @@ public void testPositionSetRowFilter() {
         .containsExactlyElementsOf(Lists.newArrayList(1L, 2L, 5L, 6L, 8L));
   }
 
-  @Test
-  public void testCombinedPositionSetRowFilter() {
+  static Stream<ExecutorService> executorServiceProvider() {
+    ExecutorService executorService =

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#discussion_r1376015003


##########
core/src/main/java/org/apache/iceberg/deletes/Deletes.java:
##########
@@ -127,14 +130,25 @@ public static StructLikeSet toEqualitySet(
 
   public static <T extends StructLike> PositionDeleteIndex toPositionIndex(
       CharSequence dataLocation, List<CloseableIterable<T>> deleteFiles) {
+    return toPositionIndex(dataLocation, deleteFiles, ThreadPools.getDeleteWorkerPool());
+  }
+
+  public static <T extends StructLike> PositionDeleteIndex toPositionIndex(
+      CharSequence dataLocation,
+      List<CloseableIterable<T>> deleteFiles,
+      ExecutorService deleteWorkerPool) {
     DataFileFilter<T> locationFilter = new DataFileFilter<>(dataLocation);
     List<CloseableIterable<Long>> positions =
         Lists.transform(
             deleteFiles,
             deletes ->
                 CloseableIterable.transform(
                     locationFilter.filter(deletes), row -> (Long) POSITION_ACCESSOR.get(row)));
-    return toPositionIndex(CloseableIterable.concat(positions));
+    if (positions.size() > 1 && (deleteWorkerPool != null)) {

Review Comment:
   ```suggestion
       if (positions.size() > 1 && deleteWorkerPool != null) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#issuecomment-1793318524

   I actually have some concerns about this change, which I outlined in [this](https://docs.google.com/document/d/1M4L6o-qnGRwGhbhkW8BnravoTwvCrJV8VvzVQDRJO5I) doc.
   I have an alternative implementation in PR #6432 that I want to consider before this change is released.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #8805:
URL: https://github.com/apache/iceberg/pull/8805


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#issuecomment-1790230032

   LGTM, thanks @wypoon 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#issuecomment-1760074698

   @rdblue @aokolnychyi as @rbalamohan indicated that he's not working on https://github.com/apache/iceberg/pull/6432 anymore, I have taken it up here. I rebased it on master and resolved the conflicts, moving the configuration from `SystemProperties` to the new `SystemConfigs`, changed the default for the pool size to be the same as for the existing worker pool, and the tests are green.
   @aokolnychyi I didn't see https://github.com/apache/iceberg/pull/6432#issuecomment-1758777424 until I have put this up. I didn't know that you're also working on this in https://github.com/apache/iceberg/pull/8755. I'll take a look at that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#discussion_r1366551865


##########
core/src/test/java/org/apache/iceberg/deletes/TestPositionFilter.java:
##########
@@ -282,6 +286,16 @@ public void testPositionSetRowFilter() {
 
   @Test
   public void testCombinedPositionSetRowFilter() {
+    testCombinedPositionSetRowFilter(null);

Review Comment:
   minor: why not make the test a `@ParameterizedTest`, taking a `null` and a configured `ExecutorService`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org