http stream as input data source

[Harald Busch <ba...@bnv-bamberg.de>]

Hi,
is there a http data stream as data source ?
I only see socketTextStream and other predefined stream sources.
It seems that I have to use fromCollection, fromElements ... and prepare the collection for myself.
Thanks
Regards

Re: http stream as input data source

[yuxia <lu...@alumni.sjtu.edu.cn>]

The quick answer is no. 
There's no http data stream on hand. 
You can implement one by yourself. Here[1] is a guidance about how to implemet user-defined source & sink 

Btw, there's a jira for http sink[2] but is marked as won't fix. 

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/ 
[2]: https://issues.apache.org/jira/browse/FLINK-8047. 

Best regards, 
Yuxia 


发件人: "Harald Busch" <ba...@bnv-bamberg.de> 
收件人: "User" <us...@flink.apache.org> 
发送时间: 星期四, 2022年 5 月 12日 上午 11:37:29 
主题: http stream as input data source 

Hi, 
is there a http data stream as data source ? 
I only see socketTextStream and other predefined stream sources. 
It seems that I have to use fromCollection, fromElements ... and prepare the collection for myself. 
Thanks 
Regards 

Re: http stream as input data source

[Alexander Preuß <al...@ververica.com>]

Hi Harald,

I was previously investigating this topic as well. There are some community
efforts for HTTP sources, please have a look at the references below:

https://getindata.com/blog/data-enrichment-flink-sql-http-connector-flink-sql-part-one/
https://github.com/getindata/flink-http-connector
https://github.com/galgus/flink-connector-http

Best regards,
Alexander

On Thu, May 12, 2022 at 1:59 PM Xuyang <xy...@163.com> wrote:

> Hi, there have not been a http source currently, and you can build the
> custom data source manually just like Yuxia said.
>
> Yuxia has given you a quick way to build the custom connector by Table
> Api. But if you want to use DataStream api to do that, you can refer to
> here[1].
>
> You can also open an issue to start a discussion in Flink community
> here[2] to let community support this feature officially.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/#use-the-source
> [2] https://issues.apache.org/jira/projects/FLINK/issues/
>
> At 2022-05-12 11:37:29, "Harald Busch" <ba...@bnv-bamberg.de> wrote:
>
> Hi,
> is there a http data stream as data source ?
> I only see socketTextStream and other predefined stream sources.
> It seems that I have to use fromCollection, fromElements ... and prepare
> the collection for myself.
> Thanks
> Regards
>
>

-- 

Alexander Preuß | Engineer - Data Intensive Systems

alexanderpreuss@ververica.com

<https://www.ververica.com/>


Follow us @VervericaData

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH

Registered at Amtsgericht Charlottenburg: HRB 158244 B

Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park Tung Jason,
Jinwei (Kevin) Zhang

Re:http stream as input data source

[Xuyang <xy...@163.com>]

Hi, there have not been a http source currently, and you can build the custom data source manually just like Yuxia said.

Yuxia has given you a quick way to build the custom connector by Table Api. But if you want to use DataStream api to do that, you can refer to here[1].

You can also open an issue to start a discussion in Flink community here[2] to let community support this feature officially.

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/#use-the-source
[2] https://issues.apache.org/jira/projects/FLINK/issues/



At 2022-05-12 11:37:29, "Harald Busch" <ba...@bnv-bamberg.de> wrote:

Hi,
is there a http data stream as data source ?
I only see socketTextStream and other predefined stream sources.
It seems that I have to use fromCollection, fromElements ... and prepare the collection for myself.
Thanks
Regards