You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Galen Warren <ga...@cvillewarrens.com> on 2022/02/02 17:30:22 UTC
Statefun async http request via RequestReplyFunctionBuilder
Is it possible to choose the async HTTP transport using
RequestReplyFunctionBuilder? It looks to me that it is not, but I wanted to
double check. Thanks.
Re: Statefun async http request via RequestReplyFunctionBuilder
Posted by Galen Warren <ga...@cvillewarrens.com>.
Ah, nevermind. I was misunderstanding how maxRequestDuration related to the
retries.
Locally, I had Flink set not to retry on failure, so once
maxRequestDuration expired without a successful result, Flink itself was
stopping, and I wasn't seeing the function get called again. But I see now
that, if Flink is set to retry on failure, then it will start from the last
checkpoint and retry, which is what I wanted to happen.
Sorry for the confusion.
On Wed, Feb 2, 2022 at 2:55 PM Galen Warren <ga...@cvillewarrens.com> wrote:
> Gotcha, thanks. I may be able to work on that one in a couple weeks if
> you're looking for help.
>
> Unrelated question -- another thing that would be useful for me would be
> the ability to set a maximum backoff interval in BoundedExponentialBackoff
> or the async equivalent. My situation is this. I'd like to set a long
> maxRequestDuration, so it takes a good while for Statefun to "give up" on a
> function call, i.e. perhaps several hours or even days. During that time,
> if the backoff interval doubles on each failure, those backoff intervals
> get pretty long.
>
> Sometimes, in exponential backoff implementations, I've seen the concept
> of a max backoff interval, i.e. once the backoff interval reaches that
> point, it won't go any higher. So I could set it to, say, 60 seconds, and
> no matter how long it would retry the function, the interval between
> retries wouldn't be more than that.
>
> Do you think that would be a useful addition? I could post something to
> the dev list if you want.
>
> On Wed, Feb 2, 2022 at 2:39 PM Igal Shilman <ig...@apache.org> wrote:
>
>> Hi Galen,
>> You are right, it is not possible, but there is no real reason for that.
>> We should fix this, and I've created the following JIRA issue [1]
>>
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-25933
>>
>> On Wed, Feb 2, 2022 at 6:30 PM Galen Warren <ga...@cvillewarrens.com>
>> wrote:
>>
>> > Is it possible to choose the async HTTP transport using
>> > RequestReplyFunctionBuilder? It looks to me that it is not, but I
>> wanted to
>> > double check. Thanks.
>> >
>>
>
Re: Statefun async http request via RequestReplyFunctionBuilder
Posted by Galen Warren <ga...@cvillewarrens.com>.
Thanks! PR created and noted on the ticket.
On Wed, Feb 9, 2022 at 3:42 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Galen,
>
> Great to hear it :-) I've assigned you to the ticket [1]. Next you can open
> a PR against the repository and we will review it.
>
> [1] https://issues.apache.org/jira/browse/FLINK-25933
>
> Cheers,
> Till
>
> On Tue, Feb 8, 2022 at 11:58 PM Galen Warren <ga...@cvillewarrens.com>
> wrote:
>
> > I'm ready to pick this one up, I have some code that's working locally.
> > Shall I create a PR?
> >
> >
> >
> > On Wed, Feb 2, 2022 at 3:17 PM Igal Shilman <ig...@gmail.com>
> > wrote:
> >
> > > Great, ping me when you would like to pick this up.
> > >
> > > For the related issue, I think that can be a good addition indeed!
> > >
> > > On Wed, Feb 2, 2022 at 8:55 PM Galen Warren <ga...@cvillewarrens.com>
> > > wrote:
> > >
> > > > Gotcha, thanks. I may be able to work on that one in a couple weeks
> if
> > > > you're looking for help.
> > > >
> > > > Unrelated question -- another thing that would be useful for me would
> > be
> > > > the ability to set a maximum backoff interval in
> > > BoundedExponentialBackoff
> > > > or the async equivalent. My situation is this. I'd like to set a long
> > > > maxRequestDuration, so it takes a good while for Statefun to "give
> up"
> > > on a
> > > > function call, i.e. perhaps several hours or even days. During that
> > time,
> > > > if the backoff interval doubles on each failure, those backoff
> > intervals
> > > > get pretty long.
> > > >
> > > > Sometimes, in exponential backoff implementations, I've seen the
> > concept
> > > of
> > > > a max backoff interval, i.e. once the backoff interval reaches that
> > > point,
> > > > it won't go any higher. So I could set it to, say, 60 seconds, and no
> > > > matter how long it would retry the function, the interval between
> > retries
> > > > wouldn't be more than that.
> > > >
> > > > Do you think that would be a useful addition? I could post something
> to
> > > the
> > > > dev list if you want.
> > > >
> > > > On Wed, Feb 2, 2022 at 2:39 PM Igal Shilman <ig...@apache.org> wrote:
> > > >
> > > > > Hi Galen,
> > > > > You are right, it is not possible, but there is no real reason for
> > > that.
> > > > > We should fix this, and I've created the following JIRA issue [1]
> > > > >
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-25933
> > > > >
> > > > > On Wed, Feb 2, 2022 at 6:30 PM Galen Warren <
> galen@cvillewarrens.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Is it possible to choose the async HTTP transport using
> > > > > > RequestReplyFunctionBuilder? It looks to me that it is not, but I
> > > > wanted
> > > > > to
> > > > > > double check. Thanks.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Statefun async http request via RequestReplyFunctionBuilder
Posted by Till Rohrmann <tr...@apache.org>.
Hi Galen,
Great to hear it :-) I've assigned you to the ticket [1]. Next you can open
a PR against the repository and we will review it.
[1] https://issues.apache.org/jira/browse/FLINK-25933
Cheers,
Till
On Tue, Feb 8, 2022 at 11:58 PM Galen Warren <ga...@cvillewarrens.com>
wrote:
> I'm ready to pick this one up, I have some code that's working locally.
> Shall I create a PR?
>
>
>
> On Wed, Feb 2, 2022 at 3:17 PM Igal Shilman <ig...@gmail.com>
> wrote:
>
> > Great, ping me when you would like to pick this up.
> >
> > For the related issue, I think that can be a good addition indeed!
> >
> > On Wed, Feb 2, 2022 at 8:55 PM Galen Warren <ga...@cvillewarrens.com>
> > wrote:
> >
> > > Gotcha, thanks. I may be able to work on that one in a couple weeks if
> > > you're looking for help.
> > >
> > > Unrelated question -- another thing that would be useful for me would
> be
> > > the ability to set a maximum backoff interval in
> > BoundedExponentialBackoff
> > > or the async equivalent. My situation is this. I'd like to set a long
> > > maxRequestDuration, so it takes a good while for Statefun to "give up"
> > on a
> > > function call, i.e. perhaps several hours or even days. During that
> time,
> > > if the backoff interval doubles on each failure, those backoff
> intervals
> > > get pretty long.
> > >
> > > Sometimes, in exponential backoff implementations, I've seen the
> concept
> > of
> > > a max backoff interval, i.e. once the backoff interval reaches that
> > point,
> > > it won't go any higher. So I could set it to, say, 60 seconds, and no
> > > matter how long it would retry the function, the interval between
> retries
> > > wouldn't be more than that.
> > >
> > > Do you think that would be a useful addition? I could post something to
> > the
> > > dev list if you want.
> > >
> > > On Wed, Feb 2, 2022 at 2:39 PM Igal Shilman <ig...@apache.org> wrote:
> > >
> > > > Hi Galen,
> > > > You are right, it is not possible, but there is no real reason for
> > that.
> > > > We should fix this, and I've created the following JIRA issue [1]
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-25933
> > > >
> > > > On Wed, Feb 2, 2022 at 6:30 PM Galen Warren <galen@cvillewarrens.com
> >
> > > > wrote:
> > > >
> > > > > Is it possible to choose the async HTTP transport using
> > > > > RequestReplyFunctionBuilder? It looks to me that it is not, but I
> > > wanted
> > > > to
> > > > > double check. Thanks.
> > > > >
> > > >
> > >
> >
>
Re: Statefun async http request via RequestReplyFunctionBuilder
Posted by Galen Warren <ga...@cvillewarrens.com>.
I'm ready to pick this one up, I have some code that's working locally.
Shall I create a PR?
On Wed, Feb 2, 2022 at 3:17 PM Igal Shilman <ig...@gmail.com> wrote:
> Great, ping me when you would like to pick this up.
>
> For the related issue, I think that can be a good addition indeed!
>
> On Wed, Feb 2, 2022 at 8:55 PM Galen Warren <ga...@cvillewarrens.com>
> wrote:
>
> > Gotcha, thanks. I may be able to work on that one in a couple weeks if
> > you're looking for help.
> >
> > Unrelated question -- another thing that would be useful for me would be
> > the ability to set a maximum backoff interval in
> BoundedExponentialBackoff
> > or the async equivalent. My situation is this. I'd like to set a long
> > maxRequestDuration, so it takes a good while for Statefun to "give up"
> on a
> > function call, i.e. perhaps several hours or even days. During that time,
> > if the backoff interval doubles on each failure, those backoff intervals
> > get pretty long.
> >
> > Sometimes, in exponential backoff implementations, I've seen the concept
> of
> > a max backoff interval, i.e. once the backoff interval reaches that
> point,
> > it won't go any higher. So I could set it to, say, 60 seconds, and no
> > matter how long it would retry the function, the interval between retries
> > wouldn't be more than that.
> >
> > Do you think that would be a useful addition? I could post something to
> the
> > dev list if you want.
> >
> > On Wed, Feb 2, 2022 at 2:39 PM Igal Shilman <ig...@apache.org> wrote:
> >
> > > Hi Galen,
> > > You are right, it is not possible, but there is no real reason for
> that.
> > > We should fix this, and I've created the following JIRA issue [1]
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-25933
> > >
> > > On Wed, Feb 2, 2022 at 6:30 PM Galen Warren <ga...@cvillewarrens.com>
> > > wrote:
> > >
> > > > Is it possible to choose the async HTTP transport using
> > > > RequestReplyFunctionBuilder? It looks to me that it is not, but I
> > wanted
> > > to
> > > > double check. Thanks.
> > > >
> > >
> >
>
Re: Statefun async http request via RequestReplyFunctionBuilder
Posted by Igal Shilman <ig...@gmail.com>.
Great, ping me when you would like to pick this up.
For the related issue, I think that can be a good addition indeed!
On Wed, Feb 2, 2022 at 8:55 PM Galen Warren <ga...@cvillewarrens.com> wrote:
> Gotcha, thanks. I may be able to work on that one in a couple weeks if
> you're looking for help.
>
> Unrelated question -- another thing that would be useful for me would be
> the ability to set a maximum backoff interval in BoundedExponentialBackoff
> or the async equivalent. My situation is this. I'd like to set a long
> maxRequestDuration, so it takes a good while for Statefun to "give up" on a
> function call, i.e. perhaps several hours or even days. During that time,
> if the backoff interval doubles on each failure, those backoff intervals
> get pretty long.
>
> Sometimes, in exponential backoff implementations, I've seen the concept of
> a max backoff interval, i.e. once the backoff interval reaches that point,
> it won't go any higher. So I could set it to, say, 60 seconds, and no
> matter how long it would retry the function, the interval between retries
> wouldn't be more than that.
>
> Do you think that would be a useful addition? I could post something to the
> dev list if you want.
>
> On Wed, Feb 2, 2022 at 2:39 PM Igal Shilman <ig...@apache.org> wrote:
>
> > Hi Galen,
> > You are right, it is not possible, but there is no real reason for that.
> > We should fix this, and I've created the following JIRA issue [1]
> >
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-25933
> >
> > On Wed, Feb 2, 2022 at 6:30 PM Galen Warren <ga...@cvillewarrens.com>
> > wrote:
> >
> > > Is it possible to choose the async HTTP transport using
> > > RequestReplyFunctionBuilder? It looks to me that it is not, but I
> wanted
> > to
> > > double check. Thanks.
> > >
> >
>
Re: Statefun async http request via RequestReplyFunctionBuilder
Posted by Galen Warren <ga...@cvillewarrens.com>.
Gotcha, thanks. I may be able to work on that one in a couple weeks if
you're looking for help.
Unrelated question -- another thing that would be useful for me would be
the ability to set a maximum backoff interval in BoundedExponentialBackoff
or the async equivalent. My situation is this. I'd like to set a long
maxRequestDuration, so it takes a good while for Statefun to "give up" on a
function call, i.e. perhaps several hours or even days. During that time,
if the backoff interval doubles on each failure, those backoff intervals
get pretty long.
Sometimes, in exponential backoff implementations, I've seen the concept of
a max backoff interval, i.e. once the backoff interval reaches that point,
it won't go any higher. So I could set it to, say, 60 seconds, and no
matter how long it would retry the function, the interval between retries
wouldn't be more than that.
Do you think that would be a useful addition? I could post something to the
dev list if you want.
On Wed, Feb 2, 2022 at 2:39 PM Igal Shilman <ig...@apache.org> wrote:
> Hi Galen,
> You are right, it is not possible, but there is no real reason for that.
> We should fix this, and I've created the following JIRA issue [1]
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-25933
>
> On Wed, Feb 2, 2022 at 6:30 PM Galen Warren <ga...@cvillewarrens.com>
> wrote:
>
> > Is it possible to choose the async HTTP transport using
> > RequestReplyFunctionBuilder? It looks to me that it is not, but I wanted
> to
> > double check. Thanks.
> >
>
Re: Statefun async http request via RequestReplyFunctionBuilder
Posted by Igal Shilman <ig...@apache.org>.
Hi Galen,
You are right, it is not possible, but there is no real reason for that.
We should fix this, and I've created the following JIRA issue [1]
[1] https://issues.apache.org/jira/browse/FLINK-25933
On Wed, Feb 2, 2022 at 6:30 PM Galen Warren <ga...@cvillewarrens.com> wrote:
> Is it possible to choose the async HTTP transport using
> RequestReplyFunctionBuilder? It looks to me that it is not, but I wanted to
> double check. Thanks.
>