[GitHub] [gobblin] ZihanLi58 edited a comment on pull request #3329: [GOBBLIN-1484]Make Gobblin metadata writer be able to support schema source DB

[GitBox <gi...@apache.org>]

ZihanLi58 edited a comment on pull request #3329:
URL: https://github.com/apache/gobblin/pull/3329#issuecomment-880262243


   > No additional comments beyond what @sv2000 mentioned above. But a broader question to deal with this kind of feature is: What should be the right way to specify "lineage" of schema between different tables? Is setting source.db in GMCE a right approach (which means you need to set this in a specific application's GMCE if you expect the application itself doesn't carry the schema during runtime, for example compaction), or is there something broader missing in the overall picture.
   
   
   Yeah I do think there is something miss broader. Ideally, we should have a source of truth relationship graph between each table, so that when we see schema update, we can modify all tables using that schema. Leveraging config store is doable, but will introduce more complexity in manage the relationships. One better way is that we can use datahub for this case, but this will need more design. As for now, I would like to support source db in the GMCE itself to make it feasible for OSS user as well
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org