You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2017/11/01 01:59:00 UTC
[jira] [Commented] (SPARK-22405) Enrich the event information and
add new event of ExternalCatalogEvent
[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233569#comment-16233569 ]
Saisai Shao commented on SPARK-22405:
-------------------------------------
Thanks [~hvanhovell] for your comments.
bq. When implementing this we just wanted to have a way to know that some metadata was about to change. A consumer could always retrieve more information about the the (to-be) changed by querying the catalog
I think this is a feasible approach to satisfy my needs. But there still requires more events to be posted, like "AlterTableEvent" or "AlterDatabaseEvent", so that user could query the catalog based on the posted events, without this user doesn't know when table/db is altered. What do you think?
> Enrich the event information and add new event of ExternalCatalogEvent
> ----------------------------------------------------------------------
>
> Key: SPARK-22405
> URL: https://issues.apache.org/jira/browse/SPARK-22405
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Saisai Shao
> Priority: Minor
>
> We're building a data lineage tool in which we need to monitor the metadata changes in {{ExternalCatalog}}, current {{ExternalCatalog}} already provides several useful events like "CreateDatabaseEvent" for custom SparkListener to use. But the information provided by such event is not rich enough, for example {{CreateTablePreEvent}} only provides "database" name and "table" name, not all the table metadata, which is hard for user to get all the table related useful information.
> So here propose to and new {{ExternalCatalogEvent}} and enrich the current existing events for all the catalog related updates.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org