You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Matt Burgess (Jira)" <ji...@apache.org> on 2023/10/25 02:46:00 UTC

[jira] [Updated] (NIFI-12130) PutIceberg: Ability to configure snapshot properties via dynamic attributes

     [ https://issues.apache.org/jira/browse/NIFI-12130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Burgess updated NIFI-12130:
--------------------------------
    Status: Patch Available  (was: Open)

> PutIceberg: Ability to configure snapshot properties via dynamic attributes
> ---------------------------------------------------------------------------
>
>                 Key: NIFI-12130
>                 URL: https://issues.apache.org/jira/browse/NIFI-12130
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Extensions
>            Reporter: William Dyson
>            Assignee: Mark Bathori
>            Priority: Minor
>              Labels: iceberg
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> *Motivation*
> Spark's implementation of Iceberg allows users to add snapshot properties, when writing data to an Iceberg table, using properties prefixed with "snapshot-property." like so:
> {{df.write}}
> {{  .option("write-format", "avro")}}
> {{  .option("snapshot-property.key", "value")}}
> {{  .insertInto("catalog.db.table") }}
> [https://iceberg.apache.org/docs/latest/spark-configuration/#write-options]
> These properties can be used to add context to Iceberg snapshots and help users locate snapshots in recovery scenarios.
> In fact, Spark automatically adds the application name as {_}spark.app.id{_}.
> Examples of when these properties might be useful include:
>  * Recording the data source used to produce the new records
>  * UUID of flow file used to update the table so it can be matched to NiFi provenance
> They can be queried from the snapshots metatable (feature of Iceberg).
> *Feature request*
> It would be great if we could configure PutIceberg to add these properties in a similar fashion (e.g. using dynamic properties of the form snapshot-property.*). Continuing with the comparison to Spark, it may also be worth automatically adding the flowfile UUID as something like {_}nifi.flowfile.id{_}.
> *Further details*
> I'm not entirely clued up on the Iceberg API, but it looks like these are set on the SnapshotUpdate (AppendFiles inherits from this class):
> [https://iceberg.apache.org/javadoc/master/org/apache/iceberg/SnapshotUpdate.html]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)