[GitHub] [gobblin] ZihanLi58 commented on a change in pull request #3324: [GOBBLIN-1485]Enable feature to get schema from writer schema when do hive registration

[GitBox <gi...@apache.org>]

ZihanLi58 commented on a change in pull request #3324:
URL: https://github.com/apache/gobblin/pull/3324#discussion_r660155720



##########
File path: gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveRegistrationUnitComparator.java
##########
@@ -142,12 +145,24 @@ public T compareIsStoredAsSubDirs() {
     return (T) this;
   }
 
+  private State extractSchemaVersion(State state) {
+    State newState = new State(state);
+    String schemaFromState = state.getProp(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName());
+    if (schemaFromState != null && !schemaFromState.isEmpty()) {
+      String schemaVersion = AvroUtils.getSchemaCreationTime(new Schema.Parser().parse(schemaFromState));
+      if (schemaVersion != null && !schemaVersion.isEmpty()) {
+         newState.removeProp(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName());

Review comment:
       The whole schema may contains special character, so when comparing whether we need to update the table, we remove this prop, but it's a new state object, so we are not removing this prop from the hive unit.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org