You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Udit Mehrotra (Jira)" <ji...@apache.org> on 2021/08/25 09:08:00 UTC
[jira] [Updated] (HUDI-1267) Additional Metadata Details for Hudi
Transactions
[ https://issues.apache.org/jira/browse/HUDI-1267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Udit Mehrotra updated HUDI-1267:
--------------------------------
Fix Version/s: (was: 0.9.0)
0.10.0
> Additional Metadata Details for Hudi Transactions
> -------------------------------------------------
>
> Key: HUDI-1267
> URL: https://issues.apache.org/jira/browse/HUDI-1267
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Usability
> Affects Versions: 0.9.0
> Reporter: Ashish M G
> Priority: Major
> Labels: features
> Fix For: 0.10.0
>
>
> Whenever following scenarios happen :
> # Custom Datasource ( Kafka for instance ) -> Hudi Table
> # Hudi -> Hudi Table
> # s3 -> Hudi Table
> Following metadata need to be captured :
> # Table Level Metadata
> *
> ** Operation name ( record level ) like Upsert, Insert etc for last operation performed on the row
> # Transaction Level Metadata ( This will be logged on Hudi Level and not Table Level )
> ** Source ( Kafka Topic Name / S3 url for source data in case of s3 etc )
> ** Target Hudi Table Name
> ** Last transaction time ( last commit time )
> Basically , point (1) collects all details on table level and point (2) collects all the transactions happened on Hudi Level
> Point(1) would be just a column addition for operation type
> Eg for Point (2) : Suppose we had an ingestion from Kafka topic 'A' to Hudi table 'ingest_kafka' and another ingestion from RDBMS table ( 'tableA' ) through Sqoop to Hudi Table 'RDBMSingest' then the metadata captured would be :
>
> |Source|Timestamp|Transaction Type|Target|
> |Kafka - 'A'|XXXXXX|UPSERT|ingest_kafka|
> |RDBMS - 'tableA'|XXXXXX|INSERT|RDBMSingest|
>
> The Transaction Details Table in Point (2) should be available as a separate common table which can be queried as Hudi Table or stored as parquet which can be queried from Spark
--
This message was sent by Atlassian Jira
(v8.3.4#803005)