You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Dennis N Brown <db...@lenovo.com> on 2020/12/10 22:27:27 UTC

Need help setting up NiFi PutS3Object processor

Running NiFi v1.12.1 , attempting to copy files from existing filesystem to S3.

Trying to set user-defined metadata on the file objects, but can't seem to figure out the correct format.

All the doc suggests it can be done with dynamic properties, so have an UpdateAttribute processor before the PutS3Object processor and have attempted everything I can think of to pass the correct format, but no metadata gets created.

I have been able to set metadata with python using something like:

s3.upload_file(Filename, bucketname, key, ExtraArgs={"Metadata": {"upload_ts":"2020-12-02 10:57:00.34343","path":"put/some/path/here"}})

so I know it can be done.



Can someone provide an example of the correct format to use.



Thanks!


Regards,

Dennis N Brown

RE: [External] Re: Need help setting up NiFi PutS3Object processor

Posted by Dennis N Brown <db...@lenovo.com>.
Hi Peter,  Thanks so much!!  That works!  I guess I had not tried that since I upgraded from v1.8 … that does not seem to work btw on v1.8.

Regards,

Dennis



From: Peter Turcsanyi <tu...@apache.org>
Sent: December 10, 2020 18:55
To: users@nifi.apache.org
Subject: [External] Re: Need help setting up NiFi PutS3Object processor

Dennis,

You can add user-defined metadata via dynamic properties of PutS3Object itself (no UpdateAttribute needed). Plus (+) icon at the top-right corner on the processor properties tab.
The name of the dynamic property will be the key (or name) of the user-defined metadata.
The value of the dynamic property will be the value of the user-defined metadata. You can use expression language for the value (eg. referencing FF attributes or now() for the timestamp).

Hope it helps.

Best,
Peter Turcsanyi

On Thu, Dec 10, 2020 at 11:27 PM Dennis N Brown <db...@lenovo.com>> wrote:

Running NiFi v1.12.1 , attempting to copy files from existing filesystem to S3.

Trying to set user-defined metadata on the file objects, but can't seem to figure out the correct format.

All the doc suggests it can be done with dynamic properties, so have an UpdateAttribute processor before the PutS3Object processor and have attempted everything I can think of to pass the correct format, but no metadata gets created.

I have been able to set metadata with python using something like:

s3.upload_file(Filename, bucketname, key, ExtraArgs={"Metadata": {"upload_ts":"2020-12-02 10:57:00.34343","path":"put/some/path/here"}})

so I know it can be done.



Can someone provide an example of the correct format to use.



Thanks!


Regards,

Dennis N Brown

Re: Need help setting up NiFi PutS3Object processor

Posted by Peter Turcsanyi <tu...@apache.org>.
Dennis,

You can add user-defined metadata via dynamic properties of PutS3Object
itself (no UpdateAttribute needed). Plus (+) icon at the top-right corner
on the processor properties tab.
The name of the dynamic property will be the key (or name) of the
user-defined metadata.
The value of the dynamic property will be the value of the user-defined
metadata. You can use expression language for the value (eg. referencing FF
attributes or now() for the timestamp).

Hope it helps.

Best,
Peter Turcsanyi

On Thu, Dec 10, 2020 at 11:27 PM Dennis N Brown <db...@lenovo.com> wrote:

> Running NiFi v1.12.1 , attempting to copy files from existing filesystem
> to S3.
>
> Trying to set user-defined metadata on the file objects, but can't seem to
> figure out the correct format.
>
> All the doc suggests it can be done with dynamic properties, so have an
> UpdateAttribute processor before the PutS3Object processor and have
> attempted everything I can think of to pass the correct format, but no
> metadata gets created.
>
> I have been able to set metadata with python using something like:
>
> *s3.upload_file(Filename, bucketname, key, ExtraArgs={"Metadata":
> {"upload_ts":"2020-12-02 10:57:00.34343","path":"put/some/path/here"}})*
>
> so I know it can be done.
>
>
>
> Can someone provide an example of the correct format to use.
>
>
>
> Thanks!
>
>
>
>
>
> Regards,
>
>
>
> Dennis N Brown
>
>
>
>
>
>
>
>
>
>
>