state and Id of job in flatMap

[anissa moussaoui <an...@dcbrain.com>]

Hello,

I created a process for an anomaly detection with a flatMap. I need to know
the end of each job at the level of the flatMap to be able to flush a
buffer in the output collector.

I saw that it is possible to get status of job by using ExecutionEnvironment,
but i don't know how i can implement it.

Is it possible to recover the end of each job in the flatMap of any
processing source with last iteration of job or to have in advance the size
of the partition that each job must deal ?

Thank's in advance !

Anissa

-- 

 <http://www.dcbrain.com/>   <https://twitter.com/dcbrain_feed?lang=fr>   
<https://www.linkedin.com/company/dcbrain>   
<https://www.youtube.com/channel/UCSJrWPBLQ58fHPN8lP_SEGw>


 Pensez à la 
planète, imprimer ce papier que si nécessaire 

Re: state and Id of job in flatMap

[anissa moussaoui <an...@dcbrain.com>]

Hello,
Thank you for your replay. I will use MapPartition for anomaly detection
for batch job. But i saw that flink has planned to unify stream and batch
according to the folowing link
https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html
, so in this case how can i use MapPartition with stream ?

Thank's in advance !

Anissa



Le mer. 20 mars 2019 à 18:53, Ken Krugler <kk...@transpac.com> a
écrit :

> Hi Anissa,
>
> I assume you’re running a batch job, since there is no “end of job” with
> streaming.
>
> If so, then calling .mapPartition(new YourMapPartitionFunction) is one
> option that I’ve used for cases like this.
>
> See MapPartitionFunction
> <https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/api/common/functions/MapPartitionFunction.html> in
> the JavaDocs.
>
> — Ken
>
>
> On Mar 20, 2019, at 8:59 AM, anissa moussaoui <
> anissa.moussaoui@dcbrain.com> wrote:
>
> Hello,
>
> I created a process for an anomaly detection with a flatMap. I need to
> know the end of each job at the level of the flatMap to be able to flush a
> buffer in the output collector.
>
> I saw that it is possible to get status of job by using ExecutionEnvironment,
> but i don't know how i can implement it.
>
> Is it possible to recover the end of each job in the flatMap of any
> processing source with last iteration of job or to have in advance the size
> of the partition that each job must deal ?
>
> Thank's in advance !
>
> Anissa
>
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>
>

-- 

 <http://www.dcbrain.com/>   <https://twitter.com/dcbrain_feed?lang=fr>   
<https://www.linkedin.com/company/dcbrain>   
<https://www.youtube.com/channel/UCSJrWPBLQ58fHPN8lP_SEGw>


 Pensez à la 
planète, imprimer ce papier que si nécessaire