You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Archit Mittal <ma...@gmail.com> on 2017/04/06 07:40:59 UTC
Connector for REST End Point
Hi
Can we write a custom connector listening to a REST Api end point in Flink
Thanks
Archit
Re: Connector for REST End Point
Posted by Archit Mittal <ma...@gmail.com>.
Hi
I wanted to know the pros and cons of both approaches and if there would be
some better approaches for the same. Thanks for the info regarding the
approaches. From your input as it has to be fault tolerant with
checkpointing i think adding my own kafka in between would be better.
Thanks
Archit
On Mon, Apr 10, 2017 at 5:42 PM, Chesnay Schepler <ch...@apache.org>
wrote:
> Hello,
>
> In general it should be possible for you to write a source that directly
> works with your REST API.
>
> The SourceFunction interface is simple enough that you can do pretty much
> anything, which makes
> me curious as to what makes you wonder whether it is possible. Have you
> tried implementing your
> own source function but hit a wall somewhere?
>
> Bear in mind though, if it has to work with checkpointing than you're
> probably better off using another
> kafka instance though.
>
> Regards,
> Chesnay
>
>
> On 10.04.2017 09:50, Archit Mittal wrote:
>
> Hi
>
> Any suggestions regarding this.
>
> Thanks
> Archit
>
> On Thu, Apr 6, 2017 at 2:52 PM, Archit Mittal <ma...@gmail.com> wrote:
>
>> Hi Chesnay
>>
>> I have a REST Api that listens to events that come through Kafka but i
>> dont have access to that Kafka . Can I write a connector that consumes
>> events through the REST end point and not directly through Kafka.
>>
>> I have a solution that i can dump the events coming to the api into
>> another Kafka instance and consume through that but it will add another hop
>> to the flow.
>>
>> Is There a better way to do this by writing a custom connector or
>> something like that .
>>
>> Thanks
>> Archit
>>
>> On Thu, Apr 6, 2017 at 1:15 PM, Chesnay Schepler < <ch...@apache.org>
>> chesnay@apache.org> wrote:
>>
>>> Hello,
>>>
>>> Please give us a bit more information, I'm really not sure what you
>>> mean. :(
>>>
>>> Regards,
>>> Chesnay
>>>
>>>
>>> On 06.04.2017 09:40, Archit Mittal wrote:
>>>
>>>> Hi
>>>>
>>>> Can we write a custom connector listening to a REST Api end point in
>>>> Flink
>>>>
>>>> Thanks
>>>> Archit
>>>>
>>>
>>>
>>
>
>
Re: Connector for REST End Point
Posted by Chesnay Schepler <ch...@apache.org>.
Hello,
In general it should be possible for you to write a source that directly
works with your REST API.
The SourceFunction interface is simple enough that you can do pretty
much anything, which makes
me curious as to what makes you wonder whether it is possible. Have you
tried implementing your
own source function but hit a wall somewhere?
Bear in mind though, if it has to work with checkpointing than you're
probably better off using another
kafka instance though.
Regards,
Chesnay
On 10.04.2017 09:50, Archit Mittal wrote:
> Hi
>
> Any suggestions regarding this.
>
> Thanks
> Archit
>
> On Thu, Apr 6, 2017 at 2:52 PM, Archit Mittal <marchit51@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Chesnay
>
> I have a REST Api that listens to events that come through Kafka
> but i dont have access to that Kafka . Can I write a connector
> that consumes events through the REST end point and not directly
> through Kafka.
>
> I have a solution that i can dump the events coming to the api
> into another Kafka instance and consume through that but it will
> add another hop to the flow.
>
> Is There a better way to do this by writing a custom connector or
> something like that .
>
> Thanks
> Archit
>
> On Thu, Apr 6, 2017 at 1:15 PM, Chesnay Schepler
> <chesnay@apache.org <ma...@apache.org>> wrote:
>
> Hello,
>
> Please give us a bit more information, I'm really not sure
> what you mean. :(
>
> Regards,
> Chesnay
>
>
> On 06.04.2017 09:40, Archit Mittal wrote:
>
> Hi
>
> Can we write a custom connector listening to a REST Api
> end point in Flink
>
> Thanks
> Archit
>
>
>
>
Re: Connector for REST End Point
Posted by Archit Mittal <ma...@gmail.com>.
Hi
Any suggestions regarding this.
Thanks
Archit
On Thu, Apr 6, 2017 at 2:52 PM, Archit Mittal <ma...@gmail.com> wrote:
> Hi Chesnay
>
> I have a REST Api that listens to events that come through Kafka but i
> dont have access to that Kafka . Can I write a connector that consumes
> events through the REST end point and not directly through Kafka.
>
> I have a solution that i can dump the events coming to the api into
> another Kafka instance and consume through that but it will add another hop
> to the flow.
>
> Is There a better way to do this by writing a custom connector or
> something like that .
>
> Thanks
> Archit
>
> On Thu, Apr 6, 2017 at 1:15 PM, Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> Hello,
>>
>> Please give us a bit more information, I'm really not sure what you mean.
>> :(
>>
>> Regards,
>> Chesnay
>>
>>
>> On 06.04.2017 09:40, Archit Mittal wrote:
>>
>>> Hi
>>>
>>> Can we write a custom connector listening to a REST Api end point in
>>> Flink
>>>
>>> Thanks
>>> Archit
>>>
>>
>>
>
Re: Connector for REST End Point
Posted by Archit Mittal <ma...@gmail.com>.
Hi Chesnay
I have a REST Api that listens to events that come through Kafka but i dont
have access to that Kafka . Can I write a connector that consumes events
through the REST end point and not directly through Kafka.
I have a solution that i can dump the events coming to the api into another
Kafka instance and consume through that but it will add another hop to the
flow.
Is There a better way to do this by writing a custom connector or something
like that .
Thanks
Archit
On Thu, Apr 6, 2017 at 1:15 PM, Chesnay Schepler <ch...@apache.org> wrote:
> Hello,
>
> Please give us a bit more information, I'm really not sure what you mean.
> :(
>
> Regards,
> Chesnay
>
>
> On 06.04.2017 09:40, Archit Mittal wrote:
>
>> Hi
>>
>> Can we write a custom connector listening to a REST Api end point in
>> Flink
>>
>> Thanks
>> Archit
>>
>
>
Re: Connector for REST End Point
Posted by Chesnay Schepler <ch...@apache.org>.
Hello,
Please give us a bit more information, I'm really not sure what you mean. :(
Regards,
Chesnay
On 06.04.2017 09:40, Archit Mittal wrote:
> Hi
>
> Can we write a custom connector listening to a REST Api end point in
> Flink
>
> Thanks
> Archit