You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Zihan Li (Jira)" <ji...@apache.org> on 2021/08/13 01:24:00 UTC

[jira] [Created] (GOBBLIN-1515) Fix hive schema not updated issue in streaming and GMIP

Zihan Li created GOBBLIN-1515:
---------------------------------

             Summary: Fix hive schema not updated issue in streaming and GMIP
                 Key: GOBBLIN-1515
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1515
             Project: Apache Gobblin
          Issue Type: Bug
            Reporter: Zihan Li


There are two issues need to be fixed:

1. Fix the issue that Hive schema is using avro schema even for orc table: Now, since we directly use avro schema literal to calculate the column and column types for orc table, it sometimes is not correct, so we need to change avro schema to orc schema and compute column types accordingly

2. Change the behavior that we only evolve avro schema literal but not the hive schema when schema evolution happen: In Streaming, when we do hive registration, we compare the current schema with schema registry and when there is schema evolution, we only update avro schema literal and leave hive schema to be lag. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)