PutS3Object submitting file many times

[Joe Gresock <jg...@gmail.com>]

Has anyone run into problems where PutS3Object submits the same file many
(hundreds) of times?  This turned out to be a problem for me since we have
bucket versioning enabled, so we have many copies of the same file.

I can't reproduce this behavior anymore in the normal flow of things, but
I'm wondering if it's possible for this to happen if the processor is
forced to administratively yield (i.e., due to disk full or too many open
files).

-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*

Re: PutS3Object submitting file many times

[Joe Skora <js...@gmail.com>]

I agree with Joe, it sounds like the NiFi commit was failing after the file
was sent.

If the processor was administratively yielding, possibly due to the disk
issues you mentioned, after completing the upload to S3, the file would be
left in the queue and picked up again for processing later, creating the
situation you observed.


On Wed, Oct 26, 2016 at 3:13 PM, Joe Witt <jo...@gmail.com> wrote:

> Joe
>
> I have not looked at this code but can say that processors are or should be
> built such that in error cases we prefer duplication over loss.  So a case
> like you describe could be a good example where session commit fails or
> something like it causing retries but in the mean time we keep sending to
> the destination.  It generally can be avoided that it would be so
> pronounced but just as a heads up I thought id call this out.
>
> Thanks
> Joe
>
> On Oct 26, 2016 9:58 AM, "Joe Gresock" <jg...@gmail.com> wrote:
>
> > Has anyone run into problems where PutS3Object submits the same file many
> > (hundreds) of times?  This turned out to be a problem for me since we
> have
> > bucket versioning enabled, so we have many copies of the same file.
> >
> > I can't reproduce this behavior anymore in the normal flow of things, but
> > I'm wondering if it's possible for this to happen if the processor is
> > forced to administratively yield (i.e., due to disk full or too many open
> > files).
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.    *-Philippians 4:12-13*
> >
>

Re: PutS3Object submitting file many times

[Joe Witt <jo...@gmail.com>]

Joe

I have not looked at this code but can say that processors are or should be
built such that in error cases we prefer duplication over loss.  So a case
like you describe could be a good example where session commit fails or
something like it causing retries but in the mean time we keep sending to
the destination.  It generally can be avoided that it would be so
pronounced but just as a heads up I thought id call this out.

Thanks
Joe

On Oct 26, 2016 9:58 AM, "Joe Gresock" <jg...@gmail.com> wrote:

> Has anyone run into problems where PutS3Object submits the same file many
> (hundreds) of times?  This turned out to be a problem for me since we have
> bucket versioning enabled, so we have many copies of the same file.
>
> I can't reproduce this behavior anymore in the normal flow of things, but
> I'm wondering if it's possible for this to happen if the processor is
> forced to administratively yield (i.e., due to disk full or too many open
> files).
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.    *-Philippians 4:12-13*
>