[jira] [Comment Edited] (FLINK-6085) flink as micro service

["Chen Qin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-6085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939209#comment-15939209 ] 

Chen Qin edited comment on FLINK-6085 at 3/24/17 8:32 PM:
----------------------------------------------------------

I would like to see if we can agree on high level first. Service is primarily rpc interaction with horizontal scalability and latency requirements.

Current way of bridge service with streaming pipeline via distributed Queue provides benefit of failure resilience and topic reuse at cost of extra hardware/software and latency, also no callback support.

[~till.rohrmann]

updates

Briefly chatted offline with Maxim, it seems a bit hard to work around distributed queue consider pipeline can restart and offset rewind anytime, loss of insertion events is not acceptable(query might be fine but seems flink already address this issue with query able states)

To echo Till's comments, yes, custom code could track those requests. Future question is if we can have a specific sink implementation which can reroute results to a specific rpc hosts (e.g http response or callback). 




was (Author: foxss):
I would like to see if we can agree on high level first. Service is primarily rpc interaction with horizontal scalability and latency requirements.

Current way of bridge service with streaming pipeline via distributed Queue provides benefit of failure resilience and topic reuse at cost of extra hardware/software and latency, also no callback support.

[~till.rohrmann]


> flink as micro service
> ----------------------
>
>                 Key: FLINK-6085
>                 URL: https://issues.apache.org/jira/browse/FLINK-6085
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataStream API, JobManager
>            Reporter: Chen Qin
>            Priority: Minor
>         Attachments: Untitled document.jpg
>
>
> Track discussion around run flink as a micro service, includes but not limited to 
> - RPC (web service endpoint) source
>   as web service endpoint accept RPC call, ingest to the streaming job(only one)
> - callback mechanism 
> - task assignment should honor deployment group (web tier hosts should be isolated from rest of task assignment)
> https://docs.google.com/document/d/1MSsTOz7xUu50dAf_8v3gsQFfJFFy9LKnULdIl26yj0o/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)