You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Ber, Jeremy" <jd...@amazon.com.INVALID> on 2022/05/17 14:21:31 UTC

[DISCUSS] FLIP-233: Introduce HTTP Connector

Hi there, We would like to start a discussion thread on FLIP-233: Introduce HTTP Connector<https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector> where we propose to create a connector for delivering arbitrary data packets from Apache Flink to an HTTP Endpoint.  This connector will give users flexibility to deliver data to any destination which supports REST endpoints. This includes REST APIs, Amazon Lambda, users internal or external consumers, among other options.

Looking forward to your feedback.

Thank you,
Jeremy Ber

Re: [DISCUSS] FLIP-233: Introduce HTTP Connector

Posted by "Ber, Jeremy" <jd...@amazon.com.INVALID>.
Hi Austin,

Thanks for the recommendations! After internal discussion I have decided to park this FLIP for now until I have more capacity to commit to it.

Jeremy

On 5/17/22, 10:26 AM, "Austin Cawley-Edwards" <au...@gmail.com> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    Hey Jeremy,

    Thanks for kicking off this discussion. As a Flink user, I too struggled
    with the lack of HTTP support and rolled my own with AsyncIO. Reading
    through the FLIP, I just have a few general questions and comments.

    * It is not clear to me if multiple HTTP methods are supported or not? It's
    listed in "Limitations" that only POSTs are allowed, but the constructor
    accepts a "method" argument.
    * More of a nit, the FLIP contains a lot of code, making it feel a bit more
    like a PR already. It would be easier to understand the proposed interfaces
    alone, and keep the implementation POC as a separate link IMO.

    Since there are so many different types of HTTP APIs, and many different
    ways of using them, I think the proposal would benefit from taking a more
    general approach to both request building and response handling. For
    instance, some APIs may return hints for retry that are not contained in
    the status code alone (e.g., a "retry-after" header or such + a 429 status
    code). Can we already think about how to more generally expose these two
    concepts? For the retries, it might be too idealistic, but standardizing on
    a retry interface like the one proposed in FLIP-232[1] would make these
    aysnc/http APIs feel much more aligned.

    wdyt?

    Austin

    [1]:
    https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963

    On Tue, May 17, 2022 at 10:21 AM Ber, Jeremy <jd...@amazon.com.invalid>
    wrote:

    > Hi there, We would like to start a discussion thread on FLIP-233:
    > Introduce HTTP Connector<
    > https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector>
    > where we propose to create a connector for delivering arbitrary data
    > packets from Apache Flink to an HTTP Endpoint.  This connector will give
    > users flexibility to deliver data to any destination which supports REST
    > endpoints. This includes REST APIs, Amazon Lambda, users internal or
    > external consumers, among other options.
    >
    > Looking forward to your feedback.
    >
    > Thank you,
    > Jeremy Ber
    >
    >

Re: [DISCUSS] FLIP-233: Introduce HTTP Connector

Posted by Austin Cawley-Edwards <au...@gmail.com>.
Hey Jeremy,

Thanks for kicking off this discussion. As a Flink user, I too struggled
with the lack of HTTP support and rolled my own with AsyncIO. Reading
through the FLIP, I just have a few general questions and comments.

* It is not clear to me if multiple HTTP methods are supported or not? It's
listed in "Limitations" that only POSTs are allowed, but the constructor
accepts a "method" argument.
* More of a nit, the FLIP contains a lot of code, making it feel a bit more
like a PR already. It would be easier to understand the proposed interfaces
alone, and keep the implementation POC as a separate link IMO.

Since there are so many different types of HTTP APIs, and many different
ways of using them, I think the proposal would benefit from taking a more
general approach to both request building and response handling. For
instance, some APIs may return hints for retry that are not contained in
the status code alone (e.g., a "retry-after" header or such + a 429 status
code). Can we already think about how to more generally expose these two
concepts? For the retries, it might be too idealistic, but standardizing on
a retry interface like the one proposed in FLIP-232[1] would make these
aysnc/http APIs feel much more aligned.

wdyt?

Austin

[1]:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963

On Tue, May 17, 2022 at 10:21 AM Ber, Jeremy <jd...@amazon.com.invalid>
wrote:

> Hi there, We would like to start a discussion thread on FLIP-233:
> Introduce HTTP Connector<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector>
> where we propose to create a connector for delivering arbitrary data
> packets from Apache Flink to an HTTP Endpoint.  This connector will give
> users flexibility to deliver data to any destination which supports REST
> endpoints. This includes REST APIs, Amazon Lambda, users internal or
> external consumers, among other options.
>
> Looking forward to your feedback.
>
> Thank you,
> Jeremy Ber
>
>