You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2020/03/27 15:28:20 UTC

[GitHub] [drill] arina-ielchiieva opened a new pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

arina-ielchiieva opened a new pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore
URL: https://github.com/apache/drill/pull/2042
 
 
   # [DRILL-7672](https://issues.apache.org/jira/browse/DRILL-7672): Make metadata type required when reading from / writing into Drill Metastore
   
   ## Description
   
   1. Upgraded Iceberg version and removed unneeded code for In / NotIn Expressions.
   2. Updated Metastore Read / Write interfaces to support required metadata types:
    a. introduced abstract Read / Write classes with boilerplate code;
    b. added delete operation with filter and metadata type;
    c. added metadata type validator which checks supported metadata types for each component;
    d. made purge operation terminal;
    e. made necessary changes in REAME.md files.
   3. Added / updated unit tests.
   
   ## Documentation
   NA
   
   ## Testing
   Added unit tests.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore
URL: https://github.com/apache/drill/pull/2042#discussion_r399614183
 
 

 ##########
 File path: metastore/iceberg-metastore/src/main/java/org/apache/drill/metastore/iceberg/components/tables/IcebergTables.java
 ##########
 @@ -42,16 +45,15 @@
  */
 public class IcebergTables implements Tables, MetastoreContext<TableMetadataUnit> {
 
-  public static final String STORAGE_PLUGIN = "storagePlugin";
-  public static final String WORKSPACE = "workspace";
-  public static final String TABLE_NAME = "tableName";
-  public static final String METADATA_KEY = "metadataKey";
-
   /**
    * Metastore Tables component partition keys, order of partitioning will be determined based
    * on order in {@link List} holder.
    */
-  private static final List<String> PARTITION_KEYS = Arrays.asList(STORAGE_PLUGIN, WORKSPACE, TABLE_NAME, METADATA_KEY);
+  private static final List<String> PARTITION_KEYS = Arrays.asList(
+    TableInfo.STORAGE_PLUGIN,
+    TableInfo.WORKSPACE,
+    TableInfo.TABLE_NAME,
+    MetadataInfo.METADATA_KEY);
 
 Review comment:
   Thanks for cleaning up this code. If these are the four types, does it make sense to define them in an enum, with the string keys as a property of the enum? Maybe as part of `MetadataType`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore
URL: https://github.com/apache/drill/pull/2042#discussion_r399689046
 
 

 ##########
 File path: metastore/iceberg-metastore/src/test/java/org/apache/drill/metastore/iceberg/components/tables/TestIcebergTablesMetastore.java
 ##########
 @@ -297,6 +302,7 @@ public void testOverwrite() {
       .workspace(tableInfo.workspace())
       .tableName(tableInfo.name())
       .metadataKey("dir0")
+      .metadataType(MetadataType.TABLE.name())
 
 Review comment:
   In `TableMetadataUnit` most of the data is stored as string to be suitable for storage (for example, in the file), since this is unit test I had to create `TableMetadataUnit` manually and thus indicate metadata type string value, in the code we usually do not do this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore
URL: https://github.com/apache/drill/pull/2042#discussion_r399614392
 
 

 ##########
 File path: metastore/iceberg-metastore/src/test/java/org/apache/drill/metastore/iceberg/components/tables/TestIcebergTablesMetastore.java
 ##########
 @@ -297,6 +302,7 @@ public void testOverwrite() {
       .workspace(tableInfo.workspace())
       .tableName(tableInfo.name())
       .metadataKey("dir0")
+      .metadataType(MetadataType.TABLE.name())
 
 Review comment:
   Some of these use the `MetadataType.TABLE`, others `MetadataType.TABLE.name()`. Do the tests make sure both work, or should they be uniform?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore
URL: https://github.com/apache/drill/pull/2042#discussion_r399688777
 
 

 ##########
 File path: metastore/iceberg-metastore/src/main/java/org/apache/drill/metastore/iceberg/components/tables/IcebergTables.java
 ##########
 @@ -42,16 +45,15 @@
  */
 public class IcebergTables implements Tables, MetastoreContext<TableMetadataUnit> {
 
-  public static final String STORAGE_PLUGIN = "storagePlugin";
-  public static final String WORKSPACE = "workspace";
-  public static final String TABLE_NAME = "tableName";
-  public static final String METADATA_KEY = "metadataKey";
-
   /**
    * Metastore Tables component partition keys, order of partitioning will be determined based
    * on order in {@link List} holder.
    */
-  private static final List<String> PARTITION_KEYS = Arrays.asList(STORAGE_PLUGIN, WORKSPACE, TABLE_NAME, METADATA_KEY);
+  private static final List<String> PARTITION_KEYS = Arrays.asList(
+    TableInfo.STORAGE_PLUGIN,
+    TableInfo.WORKSPACE,
+    TableInfo.TABLE_NAME,
+    MetadataInfo.METADATA_KEY);
 
 Review comment:
   Actually this a good idea to pull all Metastore column names into one place. I have created `MetastoreColumn` enum where all columns and their names are stored. Also made sure that in the code we refer only to `MetastoreColumn` and not to its string representation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] asfgit closed pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore
URL: https://github.com/apache/drill/pull/2042
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on issue #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore
URL: https://github.com/apache/drill/pull/2042#issuecomment-605494807
 
 
   @paul-rogers addressed code review comments. Had to do some refactoring to ensure we are always refer to `MetadataColumn` instead of its string representation, thus now PR contains slightly more files.
   @vvysotskyi you might want to take a look as well since I have touched some new areas.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on issue #2042: DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore
URL: https://github.com/apache/drill/pull/2042#issuecomment-605866287
 
 
   @vvysotskyi thanks, @paul-rogers is the PR ok to go?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services