You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/11/04 11:27:47 UTC

[GitHub] [iceberg] skambha opened a new pull request #1720: Publish incremental scan event during planning

skambha opened a new pull request #1720:
URL: https://github.com/apache/iceberg/pull/1720


   Fixes #1711 
   
   Changes include: 
   - Create a new `IncrementalScanEvent` to keep track of the `from` and `to` snapshot id
   - Publish the `IncrementalScanEvent` in` IncrementalDataTableScan` and remove the todo
   
   Tests:
   - Modify an existing test to check that the event is being published. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue merged pull request #1720: Publish incremental scan event during planning

Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1720:
URL: https://github.com/apache/iceberg/pull/1720


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] HeartSaVioR commented on pull request #1720: Publish incremental scan event during planning

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #1720:
URL: https://github.com/apache/iceberg/pull/1720#issuecomment-722197862


   You look to miss removing the TODO. Looks OK otherwise.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] skambha commented on a change in pull request #1720: Publish incremental scan event during planning

Posted by GitBox <gi...@apache.org>.
skambha commented on a change in pull request #1720:
URL: https://github.com/apache/iceberg/pull/1720#discussion_r517920351



##########
File path: core/src/test/java/org/apache/iceberg/TestIncrementalDataTableScan.java
##########
@@ -78,8 +81,28 @@ public void testAppends() {
     add(table.newAppend(), files("C"));
     add(table.newAppend(), files("D"));
     add(table.newAppend(), files("E")); // 5
+
+    class MyListener implements Listener<IncrementalScanEvent> {
+
+      IncrementalScanEvent lastEvent = null;
+
+      public void notify(IncrementalScanEvent event) {
+        this.lastEvent = event;
+      }
+
+      public IncrementalScanEvent event() {
+        return lastEvent;
+      }
+    }
+
+    MyListener listener1 = new MyListener();
+    Listeners.register(listener1, IncrementalScanEvent.class);
     filesMatch(Lists.newArrayList("B", "C", "D", "E"), appendsBetweenScan(1, 5));
+    Assert.assertTrue(listener1.event().fromSnapshotId() == 1);

Review comment:
       Thanks @HeartSaVioR!  I have updated the test to fetch and check all the fields of this event.   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] HeartSaVioR commented on a change in pull request #1720: Publish incremental scan event during planning

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #1720:
URL: https://github.com/apache/iceberg/pull/1720#discussion_r517841373



##########
File path: core/src/test/java/org/apache/iceberg/TestIncrementalDataTableScan.java
##########
@@ -78,8 +81,28 @@ public void testAppends() {
     add(table.newAppend(), files("C"));
     add(table.newAppend(), files("D"));
     add(table.newAppend(), files("E")); // 5
+
+    class MyListener implements Listener<IncrementalScanEvent> {
+
+      IncrementalScanEvent lastEvent = null;
+
+      public void notify(IncrementalScanEvent event) {
+        this.lastEvent = event;
+      }
+
+      public IncrementalScanEvent event() {
+        return lastEvent;
+      }
+    }
+
+    MyListener listener1 = new MyListener();
+    Listeners.register(listener1, IncrementalScanEvent.class);
     filesMatch(Lists.newArrayList("B", "C", "D", "E"), appendsBetweenScan(1, 5));
+    Assert.assertTrue(listener1.event().fromSnapshotId() == 1);

Review comment:
       I'm in favor of check all fields as it's the only test verifying IncrementalScanEvent, but others may have different thought on this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] skambha commented on pull request #1720: Publish incremental scan event during planning

Posted by GitBox <gi...@apache.org>.
skambha commented on pull request #1720:
URL: https://github.com/apache/iceberg/pull/1720#issuecomment-727906662


   cc @rdblue 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org