You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Tony Kurc <tr...@gmail.com> on 2015/11/24 22:05:59 UTC
PRs
All,
I was reviewing github PRs and wondering whether anyone objected to
slipping a couple that look like they're very close into 0.4.0.
NIFI-1192 (#131)
NIFI-631 (#113)
NIFI-1107 (#192)
I should have some review cycles tonight. Lots of comments on them all, and
have good "momentum".
Tony
Re: PRs
Posted by Joe Skora <js...@gmail.com>.
I gave it a quick read through and I think it looks good. It looks like
the properties are added to all the processors and the endpoint and http
client changes are set in AbstractAwsProcessor.
But that was a quick read because we're about to head out the door, I will
look at it in depth tonight.
On Thu, Nov 26, 2015 at 12:31 PM, Tony Kurc <tr...@gmail.com> wrote:
> I created NIFI-1225 which are changes to S3 processors without Multipart.
> Submitted a patch for review. Joe Skora, would you have a chance to look it
> over?
>
> On Thu, Nov 26, 2015 at 12:41 AM, Tony Kurc <tr...@gmail.com> wrote:
>
> > I'll second what joe said. non-trivial and a less than ideal API to work
> > with. seriously?! no expiry!
> >
> >
> >
> > On Thu, Nov 26, 2015 at 12:17 AM, Joe Witt <jo...@gmail.com> wrote:
> >
> >> Understood tony - thanks for digging into the review so thoroughly and
> >> Joe thank you. This is a very non-trivial contrib.
> >>
> >> On Thu, Nov 26, 2015 at 12:12 AM, Tony Kurc <tr...@gmail.com> wrote:
> >> > I recommend we push NIFI-1107 to next release. We discovered some
> unfun
> >> > issues the S3 Multipart "API" creates, notably, leaving dangling
> pieces
> >> > around [1]:
> >> >
> >> > "Once you initiate a multipart upload there is no expiry; you must
> >> > explicitly complete or abort the multipart upload"
> >> >
> >> > And charging while they're there [2]:
> >> >
> >> > "After you initiate multipart upload and upload one or more parts, you
> >> must
> >> > either complete or abort multipart upload in order to stop getting
> >> charged
> >> > for storage of the uploaded parts. Only after you either complete or
> >> abort
> >> > multipart upload, Amazon S3 frees up the parts storage and stops
> >> charging
> >> > you for the parts storage."
> >> >
> >> > This, plus grappling with persisting state for the multipart (and lack
> >> of
> >> > framework support for persistent state) means I think we have some
> more
> >> > work to do despite the feature being in a "works" state.
> >> >
> >> > [1]
> >> http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html
> >> > [2] http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
> >> >
> >> > On Wed, Nov 25, 2015 at 7:16 PM, Tony Kurc <tr...@gmail.com> wrote:
> >> >
> >> >> Things that make me feel better: The persistence mechanism is very
> >> similar
> >> >> to that of ListHDFS.
> >> >>
> >> >>
> >> >>
> >>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L417
> >> >>
> >> >> On Tue, Nov 24, 2015 at 10:56 PM, Joe Witt <jo...@gmail.com>
> wrote:
> >> >>
> >> >>> Tags are a great place to mark experimental. We used to plan for
> this
> >> >>> concept outright and make it look at scary and such on the ui. But
> >> >>> folks just didn't care. They used it anyway. Happy to revisit it
> but
> >> >>> for now perhaps just adding a tag of experimental is enough.
> >> >>>
> >> >>> If the existing code path is largely untouched then that is
> certainly
> >> >>> great for moving the ball forward. In fairness to Joe S or anyone
> >> >>> that has to persist internal process state until we offer that as
> part
> >> >>> of the framework it is much harder than we want it to be for people.
> >> >>> Will take a look through the state methods but Payne is probably the
> >> >>> best at playing the wack a mole edge case game for such things.
> >> >>>
> >> >>> On Tue, Nov 24, 2015 at 10:52 PM, Tony Kurc <tr...@gmail.com>
> wrote:
> >> >>> > So, I beat on the the patch for NIFI-1107, and as I suspected, it
> is
> >> >>> > awfully low risk for existing flows, but I think I'd need a second
> >> >>> opinion
> >> >>> > on how state is kept for resuming uploads. I believe it will work,
> >> and
> >> >>> it
> >> >>> > looks like a lot of the edge cases are covered if somehow state is
> >> lost
> >> >>> or
> >> >>> > corrupted, but I'm not sure if I am comfortable with how it fits
> >> >>> > architecturally. If someone has cycles, and can peruse the *State
> >> >>> methods
> >> >>> > (getState, persistState, ...) and weigh in, it would accelerate my
> >> >>> review
> >> >>> > significantly.
> >> >>> >
> >> >>> > Also, it sure would be great to mark features as experimental!
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <
> >> matt.c.gilman@gmail.com>
> >> >>> > wrote:
> >> >>> >
> >> >>> >> These tickets [1][2] address the incorrect validation errors we
> >> were
> >> >>> >> seeing for processors that include the Input Required annotation.
> >> These
> >> >>> >> were bugs that slipped through the NIFI-810 the review. Would be
> >> good
> >> >>> to
> >> >>> >> include if possible but I understand we need to draw the line
> >> >>> somewhere.
> >> >>> >>
> >> >>> >> As for NIFI-655, I've been struggling getting an LDAP server
> stood
> >> up
> >> >>> that
> >> >>> >> uses 2 way SSL. Hopefully we can get that squared away soon and
> >> wrap
> >> >>> this
> >> >>> >> one up. :)
> >> >>> >>
> >> >>> >> Matt
> >> >>> >>
> >> >>> >> [1] https://issues.apache.org/jira/browse/NIFI-1198
> >> >>> >> [2] https://issues.apache.org/jira/browse/NIFI-1203
> >> >>> >>
> >> >>> >> Sent from my iPhone
> >> >>> >>
> >> >>> >> > On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com>
> >> wrote:
> >> >>> >> >
> >> >>> >> > Given the testing to NIFI-1192 and review of NIFI-631 done
> >> already
> >> >>> >> > both are lower risk I think.
> >> >>> >> >
> >> >>> >> > NIFI-1107 seems very useful and helpful but we do need to be
> >> careful
> >> >>> >> > given that we know this one is already in use and this is a
> >> >>> >> > substantive change.
> >> >>> >> >
> >> >>> >> > If there are folks that can dig into review/testing of
> NIFI-1107
> >> that
> >> >>> >> > would be great. Waiting for word on NIFI-655 readiness then I
> >> think
> >> >>> >> > we should go cold and just focus on testing an RC.
> >> >>> >> >
> >> >>> >> > Thanks
> >> >>> >> > Joe
> >> >>> >> >
> >> >>> >> >> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com>
> >> >>> wrote:
> >> >>> >> >> Agreed. I know there has already been a good deal of
> discussion
> >> >>> about
> >> >>> >> >> design on all these.
> >> >>> >> >>
> >> >>> >> >>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <
> >> aldrinpiri@gmail.com
> >> >>> >
> >> >>> >> wrote:
> >> >>> >> >>>
> >> >>> >> >>> No qualms here. If they look good to go while the work and
> >> testing
> >> >>> >> >>> surrounding NIFI-655 wraps up, they might as well be
> included.
> >> >>> Would
> >> >>> >> not
> >> >>> >> >>> want to delay the release should any of these become
> >> protracted in
> >> >>> >> terms of
> >> >>> >> >>> iterations.
> >> >>> >> >>>
> >> >>> >> >>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <
> trkurc@gmail.com>
> >> >>> wrote:
> >> >>> >> >>>>
> >> >>> >> >>>> All,
> >> >>> >> >>>> I was reviewing github PRs and wondering whether anyone
> >> objected
> >> >>> to
> >> >>> >> >>>> slipping a couple that look like they're very close into
> >> 0.4.0.
> >> >>> >> >>>>
> >> >>> >> >>>> NIFI-1192 (#131)
> >> >>> >> >>>> NIFI-631 (#113)
> >> >>> >> >>>> NIFI-1107 (#192)
> >> >>> >> >>>>
> >> >>> >> >>>> I should have some review cycles tonight. Lots of comments
> on
> >> them
> >> >>> >> all,
> >> >>> >> >>> and
> >> >>> >> >>>> have good "momentum".
> >> >>> >> >>>>
> >> >>> >> >>>> Tony
> >> >>> >> >>>
> >> >>> >>
> >> >>>
> >> >>
> >> >>
> >>
> >
> >
>
Re: PRs
Posted by Tony Kurc <tr...@gmail.com>.
I created NIFI-1225 which are changes to S3 processors without Multipart.
Submitted a patch for review. Joe Skora, would you have a chance to look it
over?
On Thu, Nov 26, 2015 at 12:41 AM, Tony Kurc <tr...@gmail.com> wrote:
> I'll second what joe said. non-trivial and a less than ideal API to work
> with. seriously?! no expiry!
>
>
>
> On Thu, Nov 26, 2015 at 12:17 AM, Joe Witt <jo...@gmail.com> wrote:
>
>> Understood tony - thanks for digging into the review so thoroughly and
>> Joe thank you. This is a very non-trivial contrib.
>>
>> On Thu, Nov 26, 2015 at 12:12 AM, Tony Kurc <tr...@gmail.com> wrote:
>> > I recommend we push NIFI-1107 to next release. We discovered some unfun
>> > issues the S3 Multipart "API" creates, notably, leaving dangling pieces
>> > around [1]:
>> >
>> > "Once you initiate a multipart upload there is no expiry; you must
>> > explicitly complete or abort the multipart upload"
>> >
>> > And charging while they're there [2]:
>> >
>> > "After you initiate multipart upload and upload one or more parts, you
>> must
>> > either complete or abort multipart upload in order to stop getting
>> charged
>> > for storage of the uploaded parts. Only after you either complete or
>> abort
>> > multipart upload, Amazon S3 frees up the parts storage and stops
>> charging
>> > you for the parts storage."
>> >
>> > This, plus grappling with persisting state for the multipart (and lack
>> of
>> > framework support for persistent state) means I think we have some more
>> > work to do despite the feature being in a "works" state.
>> >
>> > [1]
>> http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html
>> > [2] http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
>> >
>> > On Wed, Nov 25, 2015 at 7:16 PM, Tony Kurc <tr...@gmail.com> wrote:
>> >
>> >> Things that make me feel better: The persistence mechanism is very
>> similar
>> >> to that of ListHDFS.
>> >>
>> >>
>> >>
>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L417
>> >>
>> >> On Tue, Nov 24, 2015 at 10:56 PM, Joe Witt <jo...@gmail.com> wrote:
>> >>
>> >>> Tags are a great place to mark experimental. We used to plan for this
>> >>> concept outright and make it look at scary and such on the ui. But
>> >>> folks just didn't care. They used it anyway. Happy to revisit it but
>> >>> for now perhaps just adding a tag of experimental is enough.
>> >>>
>> >>> If the existing code path is largely untouched then that is certainly
>> >>> great for moving the ball forward. In fairness to Joe S or anyone
>> >>> that has to persist internal process state until we offer that as part
>> >>> of the framework it is much harder than we want it to be for people.
>> >>> Will take a look through the state methods but Payne is probably the
>> >>> best at playing the wack a mole edge case game for such things.
>> >>>
>> >>> On Tue, Nov 24, 2015 at 10:52 PM, Tony Kurc <tr...@gmail.com> wrote:
>> >>> > So, I beat on the the patch for NIFI-1107, and as I suspected, it is
>> >>> > awfully low risk for existing flows, but I think I'd need a second
>> >>> opinion
>> >>> > on how state is kept for resuming uploads. I believe it will work,
>> and
>> >>> it
>> >>> > looks like a lot of the edge cases are covered if somehow state is
>> lost
>> >>> or
>> >>> > corrupted, but I'm not sure if I am comfortable with how it fits
>> >>> > architecturally. If someone has cycles, and can peruse the *State
>> >>> methods
>> >>> > (getState, persistState, ...) and weigh in, it would accelerate my
>> >>> review
>> >>> > significantly.
>> >>> >
>> >>> > Also, it sure would be great to mark features as experimental!
>> >>> >
>> >>> >
>> >>> >
>> >>> > On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <
>> matt.c.gilman@gmail.com>
>> >>> > wrote:
>> >>> >
>> >>> >> These tickets [1][2] address the incorrect validation errors we
>> were
>> >>> >> seeing for processors that include the Input Required annotation.
>> These
>> >>> >> were bugs that slipped through the NIFI-810 the review. Would be
>> good
>> >>> to
>> >>> >> include if possible but I understand we need to draw the line
>> >>> somewhere.
>> >>> >>
>> >>> >> As for NIFI-655, I've been struggling getting an LDAP server stood
>> up
>> >>> that
>> >>> >> uses 2 way SSL. Hopefully we can get that squared away soon and
>> wrap
>> >>> this
>> >>> >> one up. :)
>> >>> >>
>> >>> >> Matt
>> >>> >>
>> >>> >> [1] https://issues.apache.org/jira/browse/NIFI-1198
>> >>> >> [2] https://issues.apache.org/jira/browse/NIFI-1203
>> >>> >>
>> >>> >> Sent from my iPhone
>> >>> >>
>> >>> >> > On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com>
>> wrote:
>> >>> >> >
>> >>> >> > Given the testing to NIFI-1192 and review of NIFI-631 done
>> already
>> >>> >> > both are lower risk I think.
>> >>> >> >
>> >>> >> > NIFI-1107 seems very useful and helpful but we do need to be
>> careful
>> >>> >> > given that we know this one is already in use and this is a
>> >>> >> > substantive change.
>> >>> >> >
>> >>> >> > If there are folks that can dig into review/testing of NIFI-1107
>> that
>> >>> >> > would be great. Waiting for word on NIFI-655 readiness then I
>> think
>> >>> >> > we should go cold and just focus on testing an RC.
>> >>> >> >
>> >>> >> > Thanks
>> >>> >> > Joe
>> >>> >> >
>> >>> >> >> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com>
>> >>> wrote:
>> >>> >> >> Agreed. I know there has already been a good deal of discussion
>> >>> about
>> >>> >> >> design on all these.
>> >>> >> >>
>> >>> >> >>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <
>> aldrinpiri@gmail.com
>> >>> >
>> >>> >> wrote:
>> >>> >> >>>
>> >>> >> >>> No qualms here. If they look good to go while the work and
>> testing
>> >>> >> >>> surrounding NIFI-655 wraps up, they might as well be included.
>> >>> Would
>> >>> >> not
>> >>> >> >>> want to delay the release should any of these become
>> protracted in
>> >>> >> terms of
>> >>> >> >>> iterations.
>> >>> >> >>>
>> >>> >> >>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com>
>> >>> wrote:
>> >>> >> >>>>
>> >>> >> >>>> All,
>> >>> >> >>>> I was reviewing github PRs and wondering whether anyone
>> objected
>> >>> to
>> >>> >> >>>> slipping a couple that look like they're very close into
>> 0.4.0.
>> >>> >> >>>>
>> >>> >> >>>> NIFI-1192 (#131)
>> >>> >> >>>> NIFI-631 (#113)
>> >>> >> >>>> NIFI-1107 (#192)
>> >>> >> >>>>
>> >>> >> >>>> I should have some review cycles tonight. Lots of comments on
>> them
>> >>> >> all,
>> >>> >> >>> and
>> >>> >> >>>> have good "momentum".
>> >>> >> >>>>
>> >>> >> >>>> Tony
>> >>> >> >>>
>> >>> >>
>> >>>
>> >>
>> >>
>>
>
>
Re: PRs
Posted by Tony Kurc <tr...@gmail.com>.
I'll second what joe said. non-trivial and a less than ideal API to work
with. seriously?! no expiry!
On Thu, Nov 26, 2015 at 12:17 AM, Joe Witt <jo...@gmail.com> wrote:
> Understood tony - thanks for digging into the review so thoroughly and
> Joe thank you. This is a very non-trivial contrib.
>
> On Thu, Nov 26, 2015 at 12:12 AM, Tony Kurc <tr...@gmail.com> wrote:
> > I recommend we push NIFI-1107 to next release. We discovered some unfun
> > issues the S3 Multipart "API" creates, notably, leaving dangling pieces
> > around [1]:
> >
> > "Once you initiate a multipart upload there is no expiry; you must
> > explicitly complete or abort the multipart upload"
> >
> > And charging while they're there [2]:
> >
> > "After you initiate multipart upload and upload one or more parts, you
> must
> > either complete or abort multipart upload in order to stop getting
> charged
> > for storage of the uploaded parts. Only after you either complete or
> abort
> > multipart upload, Amazon S3 frees up the parts storage and stops charging
> > you for the parts storage."
> >
> > This, plus grappling with persisting state for the multipart (and lack of
> > framework support for persistent state) means I think we have some more
> > work to do despite the feature being in a "works" state.
> >
> > [1]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html
> > [2] http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
> >
> > On Wed, Nov 25, 2015 at 7:16 PM, Tony Kurc <tr...@gmail.com> wrote:
> >
> >> Things that make me feel better: The persistence mechanism is very
> similar
> >> to that of ListHDFS.
> >>
> >>
> >>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L417
> >>
> >> On Tue, Nov 24, 2015 at 10:56 PM, Joe Witt <jo...@gmail.com> wrote:
> >>
> >>> Tags are a great place to mark experimental. We used to plan for this
> >>> concept outright and make it look at scary and such on the ui. But
> >>> folks just didn't care. They used it anyway. Happy to revisit it but
> >>> for now perhaps just adding a tag of experimental is enough.
> >>>
> >>> If the existing code path is largely untouched then that is certainly
> >>> great for moving the ball forward. In fairness to Joe S or anyone
> >>> that has to persist internal process state until we offer that as part
> >>> of the framework it is much harder than we want it to be for people.
> >>> Will take a look through the state methods but Payne is probably the
> >>> best at playing the wack a mole edge case game for such things.
> >>>
> >>> On Tue, Nov 24, 2015 at 10:52 PM, Tony Kurc <tr...@gmail.com> wrote:
> >>> > So, I beat on the the patch for NIFI-1107, and as I suspected, it is
> >>> > awfully low risk for existing flows, but I think I'd need a second
> >>> opinion
> >>> > on how state is kept for resuming uploads. I believe it will work,
> and
> >>> it
> >>> > looks like a lot of the edge cases are covered if somehow state is
> lost
> >>> or
> >>> > corrupted, but I'm not sure if I am comfortable with how it fits
> >>> > architecturally. If someone has cycles, and can peruse the *State
> >>> methods
> >>> > (getState, persistState, ...) and weigh in, it would accelerate my
> >>> review
> >>> > significantly.
> >>> >
> >>> > Also, it sure would be great to mark features as experimental!
> >>> >
> >>> >
> >>> >
> >>> > On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <
> matt.c.gilman@gmail.com>
> >>> > wrote:
> >>> >
> >>> >> These tickets [1][2] address the incorrect validation errors we were
> >>> >> seeing for processors that include the Input Required annotation.
> These
> >>> >> were bugs that slipped through the NIFI-810 the review. Would be
> good
> >>> to
> >>> >> include if possible but I understand we need to draw the line
> >>> somewhere.
> >>> >>
> >>> >> As for NIFI-655, I've been struggling getting an LDAP server stood
> up
> >>> that
> >>> >> uses 2 way SSL. Hopefully we can get that squared away soon and wrap
> >>> this
> >>> >> one up. :)
> >>> >>
> >>> >> Matt
> >>> >>
> >>> >> [1] https://issues.apache.org/jira/browse/NIFI-1198
> >>> >> [2] https://issues.apache.org/jira/browse/NIFI-1203
> >>> >>
> >>> >> Sent from my iPhone
> >>> >>
> >>> >> > On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com>
> wrote:
> >>> >> >
> >>> >> > Given the testing to NIFI-1192 and review of NIFI-631 done already
> >>> >> > both are lower risk I think.
> >>> >> >
> >>> >> > NIFI-1107 seems very useful and helpful but we do need to be
> careful
> >>> >> > given that we know this one is already in use and this is a
> >>> >> > substantive change.
> >>> >> >
> >>> >> > If there are folks that can dig into review/testing of NIFI-1107
> that
> >>> >> > would be great. Waiting for word on NIFI-655 readiness then I
> think
> >>> >> > we should go cold and just focus on testing an RC.
> >>> >> >
> >>> >> > Thanks
> >>> >> > Joe
> >>> >> >
> >>> >> >> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com>
> >>> wrote:
> >>> >> >> Agreed. I know there has already been a good deal of discussion
> >>> about
> >>> >> >> design on all these.
> >>> >> >>
> >>> >> >>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <
> aldrinpiri@gmail.com
> >>> >
> >>> >> wrote:
> >>> >> >>>
> >>> >> >>> No qualms here. If they look good to go while the work and
> testing
> >>> >> >>> surrounding NIFI-655 wraps up, they might as well be included.
> >>> Would
> >>> >> not
> >>> >> >>> want to delay the release should any of these become protracted
> in
> >>> >> terms of
> >>> >> >>> iterations.
> >>> >> >>>
> >>> >> >>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com>
> >>> wrote:
> >>> >> >>>>
> >>> >> >>>> All,
> >>> >> >>>> I was reviewing github PRs and wondering whether anyone
> objected
> >>> to
> >>> >> >>>> slipping a couple that look like they're very close into 0.4.0.
> >>> >> >>>>
> >>> >> >>>> NIFI-1192 (#131)
> >>> >> >>>> NIFI-631 (#113)
> >>> >> >>>> NIFI-1107 (#192)
> >>> >> >>>>
> >>> >> >>>> I should have some review cycles tonight. Lots of comments on
> them
> >>> >> all,
> >>> >> >>> and
> >>> >> >>>> have good "momentum".
> >>> >> >>>>
> >>> >> >>>> Tony
> >>> >> >>>
> >>> >>
> >>>
> >>
> >>
>
Re: PRs
Posted by Joe Witt <jo...@gmail.com>.
Understood tony - thanks for digging into the review so thoroughly and
Joe thank you. This is a very non-trivial contrib.
On Thu, Nov 26, 2015 at 12:12 AM, Tony Kurc <tr...@gmail.com> wrote:
> I recommend we push NIFI-1107 to next release. We discovered some unfun
> issues the S3 Multipart "API" creates, notably, leaving dangling pieces
> around [1]:
>
> "Once you initiate a multipart upload there is no expiry; you must
> explicitly complete or abort the multipart upload"
>
> And charging while they're there [2]:
>
> "After you initiate multipart upload and upload one or more parts, you must
> either complete or abort multipart upload in order to stop getting charged
> for storage of the uploaded parts. Only after you either complete or abort
> multipart upload, Amazon S3 frees up the parts storage and stops charging
> you for the parts storage."
>
> This, plus grappling with persisting state for the multipart (and lack of
> framework support for persistent state) means I think we have some more
> work to do despite the feature being in a "works" state.
>
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html
> [2] http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
>
> On Wed, Nov 25, 2015 at 7:16 PM, Tony Kurc <tr...@gmail.com> wrote:
>
>> Things that make me feel better: The persistence mechanism is very similar
>> to that of ListHDFS.
>>
>>
>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L417
>>
>> On Tue, Nov 24, 2015 at 10:56 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>>> Tags are a great place to mark experimental. We used to plan for this
>>> concept outright and make it look at scary and such on the ui. But
>>> folks just didn't care. They used it anyway. Happy to revisit it but
>>> for now perhaps just adding a tag of experimental is enough.
>>>
>>> If the existing code path is largely untouched then that is certainly
>>> great for moving the ball forward. In fairness to Joe S or anyone
>>> that has to persist internal process state until we offer that as part
>>> of the framework it is much harder than we want it to be for people.
>>> Will take a look through the state methods but Payne is probably the
>>> best at playing the wack a mole edge case game for such things.
>>>
>>> On Tue, Nov 24, 2015 at 10:52 PM, Tony Kurc <tr...@gmail.com> wrote:
>>> > So, I beat on the the patch for NIFI-1107, and as I suspected, it is
>>> > awfully low risk for existing flows, but I think I'd need a second
>>> opinion
>>> > on how state is kept for resuming uploads. I believe it will work, and
>>> it
>>> > looks like a lot of the edge cases are covered if somehow state is lost
>>> or
>>> > corrupted, but I'm not sure if I am comfortable with how it fits
>>> > architecturally. If someone has cycles, and can peruse the *State
>>> methods
>>> > (getState, persistState, ...) and weigh in, it would accelerate my
>>> review
>>> > significantly.
>>> >
>>> > Also, it sure would be great to mark features as experimental!
>>> >
>>> >
>>> >
>>> > On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <ma...@gmail.com>
>>> > wrote:
>>> >
>>> >> These tickets [1][2] address the incorrect validation errors we were
>>> >> seeing for processors that include the Input Required annotation. These
>>> >> were bugs that slipped through the NIFI-810 the review. Would be good
>>> to
>>> >> include if possible but I understand we need to draw the line
>>> somewhere.
>>> >>
>>> >> As for NIFI-655, I've been struggling getting an LDAP server stood up
>>> that
>>> >> uses 2 way SSL. Hopefully we can get that squared away soon and wrap
>>> this
>>> >> one up. :)
>>> >>
>>> >> Matt
>>> >>
>>> >> [1] https://issues.apache.org/jira/browse/NIFI-1198
>>> >> [2] https://issues.apache.org/jira/browse/NIFI-1203
>>> >>
>>> >> Sent from my iPhone
>>> >>
>>> >> > On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
>>> >> >
>>> >> > Given the testing to NIFI-1192 and review of NIFI-631 done already
>>> >> > both are lower risk I think.
>>> >> >
>>> >> > NIFI-1107 seems very useful and helpful but we do need to be careful
>>> >> > given that we know this one is already in use and this is a
>>> >> > substantive change.
>>> >> >
>>> >> > If there are folks that can dig into review/testing of NIFI-1107 that
>>> >> > would be great. Waiting for word on NIFI-655 readiness then I think
>>> >> > we should go cold and just focus on testing an RC.
>>> >> >
>>> >> > Thanks
>>> >> > Joe
>>> >> >
>>> >> >> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com>
>>> wrote:
>>> >> >> Agreed. I know there has already been a good deal of discussion
>>> about
>>> >> >> design on all these.
>>> >> >>
>>> >> >>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <aldrinpiri@gmail.com
>>> >
>>> >> wrote:
>>> >> >>>
>>> >> >>> No qualms here. If they look good to go while the work and testing
>>> >> >>> surrounding NIFI-655 wraps up, they might as well be included.
>>> Would
>>> >> not
>>> >> >>> want to delay the release should any of these become protracted in
>>> >> terms of
>>> >> >>> iterations.
>>> >> >>>
>>> >> >>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com>
>>> wrote:
>>> >> >>>>
>>> >> >>>> All,
>>> >> >>>> I was reviewing github PRs and wondering whether anyone objected
>>> to
>>> >> >>>> slipping a couple that look like they're very close into 0.4.0.
>>> >> >>>>
>>> >> >>>> NIFI-1192 (#131)
>>> >> >>>> NIFI-631 (#113)
>>> >> >>>> NIFI-1107 (#192)
>>> >> >>>>
>>> >> >>>> I should have some review cycles tonight. Lots of comments on them
>>> >> all,
>>> >> >>> and
>>> >> >>>> have good "momentum".
>>> >> >>>>
>>> >> >>>> Tony
>>> >> >>>
>>> >>
>>>
>>
>>
Re: PRs
Posted by Tony Kurc <tr...@gmail.com>.
I recommend we push NIFI-1107 to next release. We discovered some unfun
issues the S3 Multipart "API" creates, notably, leaving dangling pieces
around [1]:
"Once you initiate a multipart upload there is no expiry; you must
explicitly complete or abort the multipart upload"
And charging while they're there [2]:
"After you initiate multipart upload and upload one or more parts, you must
either complete or abort multipart upload in order to stop getting charged
for storage of the uploaded parts. Only after you either complete or abort
multipart upload, Amazon S3 frees up the parts storage and stops charging
you for the parts storage."
This, plus grappling with persisting state for the multipart (and lack of
framework support for persistent state) means I think we have some more
work to do despite the feature being in a "works" state.
[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html
[2] http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
On Wed, Nov 25, 2015 at 7:16 PM, Tony Kurc <tr...@gmail.com> wrote:
> Things that make me feel better: The persistence mechanism is very similar
> to that of ListHDFS.
>
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L417
>
> On Tue, Nov 24, 2015 at 10:56 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Tags are a great place to mark experimental. We used to plan for this
>> concept outright and make it look at scary and such on the ui. But
>> folks just didn't care. They used it anyway. Happy to revisit it but
>> for now perhaps just adding a tag of experimental is enough.
>>
>> If the existing code path is largely untouched then that is certainly
>> great for moving the ball forward. In fairness to Joe S or anyone
>> that has to persist internal process state until we offer that as part
>> of the framework it is much harder than we want it to be for people.
>> Will take a look through the state methods but Payne is probably the
>> best at playing the wack a mole edge case game for such things.
>>
>> On Tue, Nov 24, 2015 at 10:52 PM, Tony Kurc <tr...@gmail.com> wrote:
>> > So, I beat on the the patch for NIFI-1107, and as I suspected, it is
>> > awfully low risk for existing flows, but I think I'd need a second
>> opinion
>> > on how state is kept for resuming uploads. I believe it will work, and
>> it
>> > looks like a lot of the edge cases are covered if somehow state is lost
>> or
>> > corrupted, but I'm not sure if I am comfortable with how it fits
>> > architecturally. If someone has cycles, and can peruse the *State
>> methods
>> > (getState, persistState, ...) and weigh in, it would accelerate my
>> review
>> > significantly.
>> >
>> > Also, it sure would be great to mark features as experimental!
>> >
>> >
>> >
>> > On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <ma...@gmail.com>
>> > wrote:
>> >
>> >> These tickets [1][2] address the incorrect validation errors we were
>> >> seeing for processors that include the Input Required annotation. These
>> >> were bugs that slipped through the NIFI-810 the review. Would be good
>> to
>> >> include if possible but I understand we need to draw the line
>> somewhere.
>> >>
>> >> As for NIFI-655, I've been struggling getting an LDAP server stood up
>> that
>> >> uses 2 way SSL. Hopefully we can get that squared away soon and wrap
>> this
>> >> one up. :)
>> >>
>> >> Matt
>> >>
>> >> [1] https://issues.apache.org/jira/browse/NIFI-1198
>> >> [2] https://issues.apache.org/jira/browse/NIFI-1203
>> >>
>> >> Sent from my iPhone
>> >>
>> >> > On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
>> >> >
>> >> > Given the testing to NIFI-1192 and review of NIFI-631 done already
>> >> > both are lower risk I think.
>> >> >
>> >> > NIFI-1107 seems very useful and helpful but we do need to be careful
>> >> > given that we know this one is already in use and this is a
>> >> > substantive change.
>> >> >
>> >> > If there are folks that can dig into review/testing of NIFI-1107 that
>> >> > would be great. Waiting for word on NIFI-655 readiness then I think
>> >> > we should go cold and just focus on testing an RC.
>> >> >
>> >> > Thanks
>> >> > Joe
>> >> >
>> >> >> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com>
>> wrote:
>> >> >> Agreed. I know there has already been a good deal of discussion
>> about
>> >> >> design on all these.
>> >> >>
>> >> >>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <aldrinpiri@gmail.com
>> >
>> >> wrote:
>> >> >>>
>> >> >>> No qualms here. If they look good to go while the work and testing
>> >> >>> surrounding NIFI-655 wraps up, they might as well be included.
>> Would
>> >> not
>> >> >>> want to delay the release should any of these become protracted in
>> >> terms of
>> >> >>> iterations.
>> >> >>>
>> >> >>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com>
>> wrote:
>> >> >>>>
>> >> >>>> All,
>> >> >>>> I was reviewing github PRs and wondering whether anyone objected
>> to
>> >> >>>> slipping a couple that look like they're very close into 0.4.0.
>> >> >>>>
>> >> >>>> NIFI-1192 (#131)
>> >> >>>> NIFI-631 (#113)
>> >> >>>> NIFI-1107 (#192)
>> >> >>>>
>> >> >>>> I should have some review cycles tonight. Lots of comments on them
>> >> all,
>> >> >>> and
>> >> >>>> have good "momentum".
>> >> >>>>
>> >> >>>> Tony
>> >> >>>
>> >>
>>
>
>
Re: PRs
Posted by Tony Kurc <tr...@gmail.com>.
Things that make me feel better: The persistence mechanism is very similar
to that of ListHDFS.
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L417
On Tue, Nov 24, 2015 at 10:56 PM, Joe Witt <jo...@gmail.com> wrote:
> Tags are a great place to mark experimental. We used to plan for this
> concept outright and make it look at scary and such on the ui. But
> folks just didn't care. They used it anyway. Happy to revisit it but
> for now perhaps just adding a tag of experimental is enough.
>
> If the existing code path is largely untouched then that is certainly
> great for moving the ball forward. In fairness to Joe S or anyone
> that has to persist internal process state until we offer that as part
> of the framework it is much harder than we want it to be for people.
> Will take a look through the state methods but Payne is probably the
> best at playing the wack a mole edge case game for such things.
>
> On Tue, Nov 24, 2015 at 10:52 PM, Tony Kurc <tr...@gmail.com> wrote:
> > So, I beat on the the patch for NIFI-1107, and as I suspected, it is
> > awfully low risk for existing flows, but I think I'd need a second
> opinion
> > on how state is kept for resuming uploads. I believe it will work, and it
> > looks like a lot of the edge cases are covered if somehow state is lost
> or
> > corrupted, but I'm not sure if I am comfortable with how it fits
> > architecturally. If someone has cycles, and can peruse the *State methods
> > (getState, persistState, ...) and weigh in, it would accelerate my review
> > significantly.
> >
> > Also, it sure would be great to mark features as experimental!
> >
> >
> >
> > On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <ma...@gmail.com>
> > wrote:
> >
> >> These tickets [1][2] address the incorrect validation errors we were
> >> seeing for processors that include the Input Required annotation. These
> >> were bugs that slipped through the NIFI-810 the review. Would be good to
> >> include if possible but I understand we need to draw the line somewhere.
> >>
> >> As for NIFI-655, I've been struggling getting an LDAP server stood up
> that
> >> uses 2 way SSL. Hopefully we can get that squared away soon and wrap
> this
> >> one up. :)
> >>
> >> Matt
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-1198
> >> [2] https://issues.apache.org/jira/browse/NIFI-1203
> >>
> >> Sent from my iPhone
> >>
> >> > On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
> >> >
> >> > Given the testing to NIFI-1192 and review of NIFI-631 done already
> >> > both are lower risk I think.
> >> >
> >> > NIFI-1107 seems very useful and helpful but we do need to be careful
> >> > given that we know this one is already in use and this is a
> >> > substantive change.
> >> >
> >> > If there are folks that can dig into review/testing of NIFI-1107 that
> >> > would be great. Waiting for word on NIFI-655 readiness then I think
> >> > we should go cold and just focus on testing an RC.
> >> >
> >> > Thanks
> >> > Joe
> >> >
> >> >> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com> wrote:
> >> >> Agreed. I know there has already been a good deal of discussion about
> >> >> design on all these.
> >> >>
> >> >>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com>
> >> wrote:
> >> >>>
> >> >>> No qualms here. If they look good to go while the work and testing
> >> >>> surrounding NIFI-655 wraps up, they might as well be included. Would
> >> not
> >> >>> want to delay the release should any of these become protracted in
> >> terms of
> >> >>> iterations.
> >> >>>
> >> >>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com>
> wrote:
> >> >>>>
> >> >>>> All,
> >> >>>> I was reviewing github PRs and wondering whether anyone objected to
> >> >>>> slipping a couple that look like they're very close into 0.4.0.
> >> >>>>
> >> >>>> NIFI-1192 (#131)
> >> >>>> NIFI-631 (#113)
> >> >>>> NIFI-1107 (#192)
> >> >>>>
> >> >>>> I should have some review cycles tonight. Lots of comments on them
> >> all,
> >> >>> and
> >> >>>> have good "momentum".
> >> >>>>
> >> >>>> Tony
> >> >>>
> >>
>
Re: PRs
Posted by Joe Witt <jo...@gmail.com>.
Tags are a great place to mark experimental. We used to plan for this
concept outright and make it look at scary and such on the ui. But
folks just didn't care. They used it anyway. Happy to revisit it but
for now perhaps just adding a tag of experimental is enough.
If the existing code path is largely untouched then that is certainly
great for moving the ball forward. In fairness to Joe S or anyone
that has to persist internal process state until we offer that as part
of the framework it is much harder than we want it to be for people.
Will take a look through the state methods but Payne is probably the
best at playing the wack a mole edge case game for such things.
On Tue, Nov 24, 2015 at 10:52 PM, Tony Kurc <tr...@gmail.com> wrote:
> So, I beat on the the patch for NIFI-1107, and as I suspected, it is
> awfully low risk for existing flows, but I think I'd need a second opinion
> on how state is kept for resuming uploads. I believe it will work, and it
> looks like a lot of the edge cases are covered if somehow state is lost or
> corrupted, but I'm not sure if I am comfortable with how it fits
> architecturally. If someone has cycles, and can peruse the *State methods
> (getState, persistState, ...) and weigh in, it would accelerate my review
> significantly.
>
> Also, it sure would be great to mark features as experimental!
>
>
>
> On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <ma...@gmail.com>
> wrote:
>
>> These tickets [1][2] address the incorrect validation errors we were
>> seeing for processors that include the Input Required annotation. These
>> were bugs that slipped through the NIFI-810 the review. Would be good to
>> include if possible but I understand we need to draw the line somewhere.
>>
>> As for NIFI-655, I've been struggling getting an LDAP server stood up that
>> uses 2 way SSL. Hopefully we can get that squared away soon and wrap this
>> one up. :)
>>
>> Matt
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-1198
>> [2] https://issues.apache.org/jira/browse/NIFI-1203
>>
>> Sent from my iPhone
>>
>> > On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
>> >
>> > Given the testing to NIFI-1192 and review of NIFI-631 done already
>> > both are lower risk I think.
>> >
>> > NIFI-1107 seems very useful and helpful but we do need to be careful
>> > given that we know this one is already in use and this is a
>> > substantive change.
>> >
>> > If there are folks that can dig into review/testing of NIFI-1107 that
>> > would be great. Waiting for word on NIFI-655 readiness then I think
>> > we should go cold and just focus on testing an RC.
>> >
>> > Thanks
>> > Joe
>> >
>> >> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com> wrote:
>> >> Agreed. I know there has already been a good deal of discussion about
>> >> design on all these.
>> >>
>> >>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com>
>> wrote:
>> >>>
>> >>> No qualms here. If they look good to go while the work and testing
>> >>> surrounding NIFI-655 wraps up, they might as well be included. Would
>> not
>> >>> want to delay the release should any of these become protracted in
>> terms of
>> >>> iterations.
>> >>>
>> >>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
>> >>>>
>> >>>> All,
>> >>>> I was reviewing github PRs and wondering whether anyone objected to
>> >>>> slipping a couple that look like they're very close into 0.4.0.
>> >>>>
>> >>>> NIFI-1192 (#131)
>> >>>> NIFI-631 (#113)
>> >>>> NIFI-1107 (#192)
>> >>>>
>> >>>> I should have some review cycles tonight. Lots of comments on them
>> all,
>> >>> and
>> >>>> have good "momentum".
>> >>>>
>> >>>> Tony
>> >>>
>>
Re: PRs
Posted by Tony Kurc <tr...@gmail.com>.
So, I beat on the the patch for NIFI-1107, and as I suspected, it is
awfully low risk for existing flows, but I think I'd need a second opinion
on how state is kept for resuming uploads. I believe it will work, and it
looks like a lot of the edge cases are covered if somehow state is lost or
corrupted, but I'm not sure if I am comfortable with how it fits
architecturally. If someone has cycles, and can peruse the *State methods
(getState, persistState, ...) and weigh in, it would accelerate my review
significantly.
Also, it sure would be great to mark features as experimental!
On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <ma...@gmail.com>
wrote:
> These tickets [1][2] address the incorrect validation errors we were
> seeing for processors that include the Input Required annotation. These
> were bugs that slipped through the NIFI-810 the review. Would be good to
> include if possible but I understand we need to draw the line somewhere.
>
> As for NIFI-655, I've been struggling getting an LDAP server stood up that
> uses 2 way SSL. Hopefully we can get that squared away soon and wrap this
> one up. :)
>
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-1198
> [2] https://issues.apache.org/jira/browse/NIFI-1203
>
> Sent from my iPhone
>
> > On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
> >
> > Given the testing to NIFI-1192 and review of NIFI-631 done already
> > both are lower risk I think.
> >
> > NIFI-1107 seems very useful and helpful but we do need to be careful
> > given that we know this one is already in use and this is a
> > substantive change.
> >
> > If there are folks that can dig into review/testing of NIFI-1107 that
> > would be great. Waiting for word on NIFI-655 readiness then I think
> > we should go cold and just focus on testing an RC.
> >
> > Thanks
> > Joe
> >
> >> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com> wrote:
> >> Agreed. I know there has already been a good deal of discussion about
> >> design on all these.
> >>
> >>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com>
> wrote:
> >>>
> >>> No qualms here. If they look good to go while the work and testing
> >>> surrounding NIFI-655 wraps up, they might as well be included. Would
> not
> >>> want to delay the release should any of these become protracted in
> terms of
> >>> iterations.
> >>>
> >>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
> >>>>
> >>>> All,
> >>>> I was reviewing github PRs and wondering whether anyone objected to
> >>>> slipping a couple that look like they're very close into 0.4.0.
> >>>>
> >>>> NIFI-1192 (#131)
> >>>> NIFI-631 (#113)
> >>>> NIFI-1107 (#192)
> >>>>
> >>>> I should have some review cycles tonight. Lots of comments on them
> all,
> >>> and
> >>>> have good "momentum".
> >>>>
> >>>> Tony
> >>>
>
Re: PRs
Posted by Matt Gilman <ma...@gmail.com>.
Tomorrow I will update the documentation for NIFI-655 and let the PR commence. While that is ongoing I'll cycle back to testing this configuration.
Matt
Sent from my iPhone
> On Nov 24, 2015, at 10:44 PM, Joe Witt <jo...@gmail.com> wrote:
>
> For 1198 and 1203 those should be included. Those are the sorts of
> things we're going to be finding/fixing in the testing phase anyway.
> Also, since they're impacting an as of yet unreleased feature we
> really need the multiple eyes on testing which is how these were
> found.
>
> For NIFI-655 why not simply document that it is an untested
> configuration and move forward?
>
> Thanks
> Joe
>
>> On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <ma...@gmail.com> wrote:
>> These tickets [1][2] address the incorrect validation errors we were seeing for processors that include the Input Required annotation. These were bugs that slipped through the NIFI-810 the review. Would be good to include if possible but I understand we need to draw the line somewhere.
>>
>> As for NIFI-655, I've been struggling getting an LDAP server stood up that uses 2 way SSL. Hopefully we can get that squared away soon and wrap this one up. :)
>>
>> Matt
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-1198
>> [2] https://issues.apache.org/jira/browse/NIFI-1203
>>
>> Sent from my iPhone
>>
>>> On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
>>>
>>> Given the testing to NIFI-1192 and review of NIFI-631 done already
>>> both are lower risk I think.
>>>
>>> NIFI-1107 seems very useful and helpful but we do need to be careful
>>> given that we know this one is already in use and this is a
>>> substantive change.
>>>
>>> If there are folks that can dig into review/testing of NIFI-1107 that
>>> would be great. Waiting for word on NIFI-655 readiness then I think
>>> we should go cold and just focus on testing an RC.
>>>
>>> Thanks
>>> Joe
>>>
>>>> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com> wrote:
>>>> Agreed. I know there has already been a good deal of discussion about
>>>> design on all these.
>>>>
>>>>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com> wrote:
>>>>>
>>>>> No qualms here. If they look good to go while the work and testing
>>>>> surrounding NIFI-655 wraps up, they might as well be included. Would not
>>>>> want to delay the release should any of these become protracted in terms of
>>>>> iterations.
>>>>>
>>>>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
>>>>>>
>>>>>> All,
>>>>>> I was reviewing github PRs and wondering whether anyone objected to
>>>>>> slipping a couple that look like they're very close into 0.4.0.
>>>>>>
>>>>>> NIFI-1192 (#131)
>>>>>> NIFI-631 (#113)
>>>>>> NIFI-1107 (#192)
>>>>>>
>>>>>> I should have some review cycles tonight. Lots of comments on them all,
>>>>> and
>>>>>> have good "momentum".
>>>>>>
>>>>>> Tony
>>>>>
Re: PRs
Posted by Joe Witt <jo...@gmail.com>.
For 1198 and 1203 those should be included. Those are the sorts of
things we're going to be finding/fixing in the testing phase anyway.
Also, since they're impacting an as of yet unreleased feature we
really need the multiple eyes on testing which is how these were
found.
For NIFI-655 why not simply document that it is an untested
configuration and move forward?
Thanks
Joe
On Tue, Nov 24, 2015 at 10:36 PM, Matt Gilman <ma...@gmail.com> wrote:
> These tickets [1][2] address the incorrect validation errors we were seeing for processors that include the Input Required annotation. These were bugs that slipped through the NIFI-810 the review. Would be good to include if possible but I understand we need to draw the line somewhere.
>
> As for NIFI-655, I've been struggling getting an LDAP server stood up that uses 2 way SSL. Hopefully we can get that squared away soon and wrap this one up. :)
>
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-1198
> [2] https://issues.apache.org/jira/browse/NIFI-1203
>
> Sent from my iPhone
>
>> On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Given the testing to NIFI-1192 and review of NIFI-631 done already
>> both are lower risk I think.
>>
>> NIFI-1107 seems very useful and helpful but we do need to be careful
>> given that we know this one is already in use and this is a
>> substantive change.
>>
>> If there are folks that can dig into review/testing of NIFI-1107 that
>> would be great. Waiting for word on NIFI-655 readiness then I think
>> we should go cold and just focus on testing an RC.
>>
>> Thanks
>> Joe
>>
>>> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com> wrote:
>>> Agreed. I know there has already been a good deal of discussion about
>>> design on all these.
>>>
>>>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com> wrote:
>>>>
>>>> No qualms here. If they look good to go while the work and testing
>>>> surrounding NIFI-655 wraps up, they might as well be included. Would not
>>>> want to delay the release should any of these become protracted in terms of
>>>> iterations.
>>>>
>>>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
>>>>>
>>>>> All,
>>>>> I was reviewing github PRs and wondering whether anyone objected to
>>>>> slipping a couple that look like they're very close into 0.4.0.
>>>>>
>>>>> NIFI-1192 (#131)
>>>>> NIFI-631 (#113)
>>>>> NIFI-1107 (#192)
>>>>>
>>>>> I should have some review cycles tonight. Lots of comments on them all,
>>>> and
>>>>> have good "momentum".
>>>>>
>>>>> Tony
>>>>
Re: PRs
Posted by Matt Gilman <ma...@gmail.com>.
These tickets [1][2] address the incorrect validation errors we were seeing for processors that include the Input Required annotation. These were bugs that slipped through the NIFI-810 the review. Would be good to include if possible but I understand we need to draw the line somewhere.
As for NIFI-655, I've been struggling getting an LDAP server stood up that uses 2 way SSL. Hopefully we can get that squared away soon and wrap this one up. :)
Matt
[1] https://issues.apache.org/jira/browse/NIFI-1198
[2] https://issues.apache.org/jira/browse/NIFI-1203
Sent from my iPhone
> On Nov 24, 2015, at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
>
> Given the testing to NIFI-1192 and review of NIFI-631 done already
> both are lower risk I think.
>
> NIFI-1107 seems very useful and helpful but we do need to be careful
> given that we know this one is already in use and this is a
> substantive change.
>
> If there are folks that can dig into review/testing of NIFI-1107 that
> would be great. Waiting for word on NIFI-655 readiness then I think
> we should go cold and just focus on testing an RC.
>
> Thanks
> Joe
>
>> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com> wrote:
>> Agreed. I know there has already been a good deal of discussion about
>> design on all these.
>>
>>> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com> wrote:
>>>
>>> No qualms here. If they look good to go while the work and testing
>>> surrounding NIFI-655 wraps up, they might as well be included. Would not
>>> want to delay the release should any of these become protracted in terms of
>>> iterations.
>>>
>>>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
>>>>
>>>> All,
>>>> I was reviewing github PRs and wondering whether anyone objected to
>>>> slipping a couple that look like they're very close into 0.4.0.
>>>>
>>>> NIFI-1192 (#131)
>>>> NIFI-631 (#113)
>>>> NIFI-1107 (#192)
>>>>
>>>> I should have some review cycles tonight. Lots of comments on them all,
>>> and
>>>> have good "momentum".
>>>>
>>>> Tony
>>>
Re: PRs
Posted by Joe Witt <jo...@gmail.com>.
understood - thanks.
On Tue, Nov 24, 2015 at 10:28 PM, Tony Kurc <tr...@gmail.com> wrote:
> Joe - I reviewed NIFI-1107 when it was a processor on its own
> (PutS3Multipart) and have an S3 testbed set up. I'm working through it now.
> I believe Joe S. has been stress testing it as well. The good news is that
> the code path for small files to S3 is largely unmolested, so it is not as
> high risk as you might think.
>
> On Tue, Nov 24, 2015 at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Given the testing to NIFI-1192 and review of NIFI-631 done already
>> both are lower risk I think.
>>
>> NIFI-1107 seems very useful and helpful but we do need to be careful
>> given that we know this one is already in use and this is a
>> substantive change.
>>
>> If there are folks that can dig into review/testing of NIFI-1107 that
>> would be great. Waiting for word on NIFI-655 readiness then I think
>> we should go cold and just focus on testing an RC.
>>
>> Thanks
>> Joe
>>
>> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com> wrote:
>> > Agreed. I know there has already been a good deal of discussion about
>> > design on all these.
>> >
>> > On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com>
>> wrote:
>> >
>> >> No qualms here. If they look good to go while the work and testing
>> >> surrounding NIFI-655 wraps up, they might as well be included. Would not
>> >> want to delay the release should any of these become protracted in
>> terms of
>> >> iterations.
>> >>
>> >> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
>> >>
>> >> > All,
>> >> > I was reviewing github PRs and wondering whether anyone objected to
>> >> > slipping a couple that look like they're very close into 0.4.0.
>> >> >
>> >> > NIFI-1192 (#131)
>> >> > NIFI-631 (#113)
>> >> > NIFI-1107 (#192)
>> >> >
>> >> > I should have some review cycles tonight. Lots of comments on them
>> all,
>> >> and
>> >> > have good "momentum".
>> >> >
>> >> > Tony
>> >> >
>> >>
>>
Re: PRs
Posted by Tony Kurc <tr...@gmail.com>.
Joe - I reviewed NIFI-1107 when it was a processor on its own
(PutS3Multipart) and have an S3 testbed set up. I'm working through it now.
I believe Joe S. has been stress testing it as well. The good news is that
the code path for small files to S3 is largely unmolested, so it is not as
high risk as you might think.
On Tue, Nov 24, 2015 at 10:23 PM, Joe Witt <jo...@gmail.com> wrote:
> Given the testing to NIFI-1192 and review of NIFI-631 done already
> both are lower risk I think.
>
> NIFI-1107 seems very useful and helpful but we do need to be careful
> given that we know this one is already in use and this is a
> substantive change.
>
> If there are folks that can dig into review/testing of NIFI-1107 that
> would be great. Waiting for word on NIFI-655 readiness then I think
> we should go cold and just focus on testing an RC.
>
> Thanks
> Joe
>
> On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com> wrote:
> > Agreed. I know there has already been a good deal of discussion about
> > design on all these.
> >
> > On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com>
> wrote:
> >
> >> No qualms here. If they look good to go while the work and testing
> >> surrounding NIFI-655 wraps up, they might as well be included. Would not
> >> want to delay the release should any of these become protracted in
> terms of
> >> iterations.
> >>
> >> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
> >>
> >> > All,
> >> > I was reviewing github PRs and wondering whether anyone objected to
> >> > slipping a couple that look like they're very close into 0.4.0.
> >> >
> >> > NIFI-1192 (#131)
> >> > NIFI-631 (#113)
> >> > NIFI-1107 (#192)
> >> >
> >> > I should have some review cycles tonight. Lots of comments on them
> all,
> >> and
> >> > have good "momentum".
> >> >
> >> > Tony
> >> >
> >>
>
Re: PRs
Posted by Joe Witt <jo...@gmail.com>.
Given the testing to NIFI-1192 and review of NIFI-631 done already
both are lower risk I think.
NIFI-1107 seems very useful and helpful but we do need to be careful
given that we know this one is already in use and this is a
substantive change.
If there are folks that can dig into review/testing of NIFI-1107 that
would be great. Waiting for word on NIFI-655 readiness then I think
we should go cold and just focus on testing an RC.
Thanks
Joe
On Tue, Nov 24, 2015 at 4:22 PM, Tony Kurc <tr...@gmail.com> wrote:
> Agreed. I know there has already been a good deal of discussion about
> design on all these.
>
> On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com> wrote:
>
>> No qualms here. If they look good to go while the work and testing
>> surrounding NIFI-655 wraps up, they might as well be included. Would not
>> want to delay the release should any of these become protracted in terms of
>> iterations.
>>
>> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
>>
>> > All,
>> > I was reviewing github PRs and wondering whether anyone objected to
>> > slipping a couple that look like they're very close into 0.4.0.
>> >
>> > NIFI-1192 (#131)
>> > NIFI-631 (#113)
>> > NIFI-1107 (#192)
>> >
>> > I should have some review cycles tonight. Lots of comments on them all,
>> and
>> > have good "momentum".
>> >
>> > Tony
>> >
>>
Re: PRs
Posted by Tony Kurc <tr...@gmail.com>.
Agreed. I know there has already been a good deal of discussion about
design on all these.
On Tue, Nov 24, 2015 at 4:14 PM, Aldrin Piri <al...@gmail.com> wrote:
> No qualms here. If they look good to go while the work and testing
> surrounding NIFI-655 wraps up, they might as well be included. Would not
> want to delay the release should any of these become protracted in terms of
> iterations.
>
> On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
>
> > All,
> > I was reviewing github PRs and wondering whether anyone objected to
> > slipping a couple that look like they're very close into 0.4.0.
> >
> > NIFI-1192 (#131)
> > NIFI-631 (#113)
> > NIFI-1107 (#192)
> >
> > I should have some review cycles tonight. Lots of comments on them all,
> and
> > have good "momentum".
> >
> > Tony
> >
>
Re: PRs
Posted by Aldrin Piri <al...@gmail.com>.
No qualms here. If they look good to go while the work and testing
surrounding NIFI-655 wraps up, they might as well be included. Would not
want to delay the release should any of these become protracted in terms of
iterations.
On Tue, Nov 24, 2015 at 4:05 PM, Tony Kurc <tr...@gmail.com> wrote:
> All,
> I was reviewing github PRs and wondering whether anyone objected to
> slipping a couple that look like they're very close into 0.4.0.
>
> NIFI-1192 (#131)
> NIFI-631 (#113)
> NIFI-1107 (#192)
>
> I should have some review cycles tonight. Lots of comments on them all, and
> have good "momentum".
>
> Tony
>