You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Will-Lo (via GitHub)" <gi...@apache.org> on 2024/09/13 23:18:54 UTC

[PR] [GOBBLIN-2157] Copy table properties in iceberg distcp [gobblin]

Will-Lo opened a new pull request, #4056:
URL: https://github.com/apache/gobblin/pull/4056

   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-2157
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   The existing iceberg registration step has the following snippet:
   ```
       if (dstMetadata != null) {
         // use current destination metadata as 'base metadata' and source as 'updated metadata' while committing
         this.tableOps.commit(dstMetadata, srcMetadata.replaceProperties(dstMetadata.properties())); <<<<<------- BUG
       }
   ```
   Since Iceberg's `replaceProperties` https://github.com/apache/iceberg/blob/e449d3405cfdb304c94835845bd8f34a73b4a517/core/src/main/java/org/apache/iceberg/TableMetadata.java#L583 just replaces the full set of properties, not just any existing properties, the result is that all of the properties are equivalent to the destination table metadata properties. Which effectively copies no properties.
   
   This PR modifies this behavior so that the destination table properties can be overwritten by the source table properties if the source table updates existing properties, as well as maintaining any existing properties in the destination that is not defined on the source.
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   Unit tests
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2157] Copy table properties in iceberg distcp [gobblin]

Posted by "Blazer-007 (via GitHub)" <gi...@apache.org>.
Blazer-007 commented on PR #4056:
URL: https://github.com/apache/gobblin/pull/4056#issuecomment-2350865499

   @Will-Lo I believe if we want to copy some source side table properties then we should copy only selective property not all,
    
   As lets say we change some property like clusterId or tableId or any other and that property is copied over to destination but destination requires this doesn't to be changed then in this case we will eventually be corrupting the properties.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2157] Copy table properties in iceberg distcp [gobblin]

Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #4056:
URL: https://github.com/apache/gobblin/pull/4056#discussion_r1762088183


##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergTable.java:
##########
@@ -212,8 +212,9 @@ public DatasetDescriptor getDatasetDescriptor(FileSystem fs) {
    * @param dstMetadata is null if destination {@link IcebergTable} is absent, in which case registration is skipped */
   protected void registerIcebergTable(TableMetadata srcMetadata, TableMetadata dstMetadata) {
     if (dstMetadata != null) {
-      // use current destination metadata as 'base metadata' and source as 'updated metadata' while committing
-      this.tableOps.commit(dstMetadata, srcMetadata.replaceProperties(dstMetadata.properties()));
+      // Use current destination metadata as 'base metadata' and but update to source metadata when committing
+      // If any of the source table properties are deleted, they will be reflected in the destination table

Review Comment:
   "and but"
   
   how about - 
   > ...as 'base metadata', but commit the source-side metadata, to synchronize source-side property deletion over to the destination
   
   ?



##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergTableTest.java:
##########
@@ -181,6 +184,33 @@ public void testGetIncrementalSnapshotInfosIteratorRepeatedFiles() throws IOExce
     }
   }
 
+  /** Verify that registerIcebergTable will update existing table properties */
+  @Test
+  public void testNewTablePropertiesAreRegistered() throws Exception {
+    Map<String, String> srcTableProperties = Maps.newHashMap();
+    Map<String, String> destTableProperties = Maps.newHashMap();
+
+    srcTableProperties.put("newKey", "newValue");
+    // Expect the old value to be overwritten by the new value
+    srcTableProperties.put("testKey", "testValueNew");
+    destTableProperties.put("testKey", "testValueOld");
+    // Expect existing property values to be maintained if it does not exist on the source

Review Comment:
   comment looks out-of-date



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2157] Copy table properties in iceberg distcp [gobblin]

Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #4056:
URL: https://github.com/apache/gobblin/pull/4056#discussion_r1759607239


##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergTable.java:
##########
@@ -213,7 +215,10 @@ public DatasetDescriptor getDatasetDescriptor(FileSystem fs) {
   protected void registerIcebergTable(TableMetadata srcMetadata, TableMetadata dstMetadata) {
     if (dstMetadata != null) {
       // use current destination metadata as 'base metadata' and source as 'updated metadata' while committing
-      this.tableOps.commit(dstMetadata, srcMetadata.replaceProperties(dstMetadata.properties()));
+      Map<String, String> combinedMetadataProperties = Maps.newHashMap();
+      combinedMetadataProperties.putAll(dstMetadata.properties());
+      combinedMetadataProperties.putAll(srcMetadata.properties());

Review Comment:
   please add a comment that captures the reasoning behind overshadowing dest with src



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2157] Copy table properties in iceberg distcp [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo merged PR #4056:
URL: https://github.com/apache/gobblin/pull/4056


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2157] Copy table properties in iceberg distcp [gobblin]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #4056:
URL: https://github.com/apache/gobblin/pull/4056#issuecomment-2350725879

   ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4056?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 41.15%. Comparing base [(`45ad13e`)](https://app.codecov.io/gh/apache/gobblin/commit/45ad13e07ebe212002b279132e1fa4d121a9294f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`aa4e35e`)](https://app.codecov.io/gh/apache/gobblin/commit/aa4e35eb98312a765fdf86c7f0e78c873466c705?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 7 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #4056      +/-   ##
   ============================================
   - Coverage     45.12%   41.15%   -3.97%     
   + Complexity     3199     2220     -979     
   ============================================
     Files           705      483     -222     
     Lines         26949    20490    -6459     
     Branches       2680     2373     -307     
   ============================================
   - Hits          12160     8433    -3727     
   + Misses        13781    11157    -2624     
   + Partials       1008      900     -108     
   ```
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/gobblin/pull/4056?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org