Re: ML/DL via Flink

["m@xi" <ma...@gmail.com>]

Hello Timo and Bechet,

@Timo: Thanks a lot for the message forwarding.

@Bechet: Thanks for your answer. I was not aware of *flink-ai-extended*
project.

Also I was not aware of the fact that ALink is striving to become the new
FlinkML. Definitely, I will look into ALink and flink-ai-extended. Alibaba
knows better haha. 

To address your answer:

"I guess it depends on what exactly you want to do. If you are doing a
training running for hours with a lot of rounds of iterations until it
converges, having it trained separately and then porting it to Flink for
inference might not lose too much efficiency. However, if you are doing
online learning to incrementally update your model as the samples flow by,
having such incremental training embedded into Flink would make a lot of
sense. Flink-ai-extended was created to support both cases, but it is
definitely more attractive in the incremental training case."

--> Well, since I focus on streaming I think an online training and serving
solution, a.k.a. prequential training, would be more suitable. There are two
problems I see in this direction though:
1) To enable *efficient* and *reliable* (super important as well) training
in a streaming fashion, Flink DataStream part should support iterations
thoroughly. However, I have read from multiple sources that iterations on
Flink are not there yet. This is why I searched for other solutions to
investigate what they do. Alibaba seems like a good direction as you said
also ALink is used in production.

2) Due to lack of reliability, in the majority of use cases prequential
training is not chosen by companies, which instead rely on the solution you
described; hence, (re)training the model (maybe for hours) and port it to
Flink whenever it is ready.

Nevertheless, thanks a lot for your answers. 

@Flinkers: Let's gather any other solutions that exist and are not listed. 

Best,
Max



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/