You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Mike S <88...@gmail.com> on 2020/02/03 12:51:05 UTC

PutParquet Deleted ALL S3 Folders

I have a PutParquet (NiFi 1.10.0) processor that was working just fine in
previous versions.  It was running earlier this week and deleted all the
data in the S3 path.  Fortunately we had versioning turned on.

With a S3 folder path of:

folder1/folder2/folder3/folder4/parq/filename

For some reason it deleted (BATCH.DELETE) folder1.  I can clearly see in
the S3 audit log that the instance role on a node in the NiFi cluster
deleted that folder.

I'm not sure if this is an eventual consistency with S3 issue.  I see many
"NO SUCH KEY" errors in the S3 audit log file when trying to delete the
original dot filename.  The no key error could also be because it deleted
folder1.

I'm testing this in a separate S3 folder (same bucket) to see if it happens
again.

Any ideas?