You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/10/26 23:26:00 UTC

[jira] [Commented] (NIFI-12130) PutIceberg: Ability to configure snapshot properties via dynamic attributes

    [ https://issues.apache.org/jira/browse/NIFI-12130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17780122#comment-17780122 ] 

ASF subversion and git services commented on NIFI-12130:
--------------------------------------------------------

Commit 67767b6550ccce6dceae7aeb61b6913bbd2afb34 in nifi's branch refs/heads/support/nifi-1.x from Mark Bathori
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=67767b6550 ]

NIFI-12130: Ability to configure snapshot properties via dynamic attributes in PutIceberg

Fix dynamic field expression language handling

Signed-off-by: Matt Burgess <ma...@apache.org>


> PutIceberg: Ability to configure snapshot properties via dynamic attributes
> ---------------------------------------------------------------------------
>
>                 Key: NIFI-12130
>                 URL: https://issues.apache.org/jira/browse/NIFI-12130
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Extensions
>            Reporter: William Dyson
>            Assignee: Mark Bathori
>            Priority: Minor
>              Labels: iceberg
>             Fix For: 1.latest, 2.latest
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> *Motivation*
> Spark's implementation of Iceberg allows users to add snapshot properties, when writing data to an Iceberg table, using properties prefixed with "snapshot-property." like so:
> {{df.write}}
> {{  .option("write-format", "avro")}}
> {{  .option("snapshot-property.key", "value")}}
> {{  .insertInto("catalog.db.table") }}
> [https://iceberg.apache.org/docs/latest/spark-configuration/#write-options]
> These properties can be used to add context to Iceberg snapshots and help users locate snapshots in recovery scenarios.
> In fact, Spark automatically adds the application name as {_}spark.app.id{_}.
> Examples of when these properties might be useful include:
>  * Recording the data source used to produce the new records
>  * UUID of flow file used to update the table so it can be matched to NiFi provenance
> They can be queried from the snapshots metatable (feature of Iceberg).
> *Feature request*
> It would be great if we could configure PutIceberg to add these properties in a similar fashion (e.g. using dynamic properties of the form snapshot-property.*). Continuing with the comparison to Spark, it may also be worth automatically adding the flowfile UUID as something like {_}nifi.flowfile.id{_}.
> *Further details*
> I'm not entirely clued up on the Iceberg API, but it looks like these are set on the SnapshotUpdate (AppendFiles inherits from this class):
> [https://iceberg.apache.org/javadoc/master/org/apache/iceberg/SnapshotUpdate.html]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)