You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Flavio Pompermaier <po...@okkam.it> on 2015/05/26 17:58:35 UTC
Dataset output callback
Hi to all,
In my program I'd like to infer from a mysql table the list of directory I
have to output on hdfs (status=0).
Once my job finish to clean each directory and update the status value of
my sql table.
How can I do that in Flink? Is there any callback on the dataset.output()
finish?
Best,
Flavio
Re: Dataset output callback
Posted by Flavio Pompermaier <po...@okkam.it>.
I'll look into that! Thanks for the moment
On Wed, May 27, 2015 at 9:24 AM, Stephan Ewen <se...@apache.org> wrote:
> If you just want to do something after the job finishes, just put the
> finalization code after the call to "execute()":
>
> env.execute();
> myCustomOutputFinalization();
>
> If you want each parallel output to do something after it finished, then
> override the "close()" method of the OutputFormat.
>
> To have a finalization call on the JobManager, make sure the OutputFormat
> implements "FinalizeOnMaster" and override that interface's method.
>
> Does any of that work for you?
> Am 27.05.2015 08:43 schrieb "Flavio Pompermaier" <po...@okkam.it>:
>
>> Any insight about this?
>>
>> On Tue, May 26, 2015 at 5:58 PM, Flavio Pompermaier <pompermaier@okkam.it
>> > wrote:
>>
>>> Hi to all,
>>>
>>> In my program I'd like to infer from a mysql table the list of directory
>>> I have to output on hdfs (status=0).
>>> Once my job finish to clean each directory and update the status value
>>> of my sql table.
>>> How can I do that in Flink? Is there any callback on the
>>> dataset.output() finish?
>>>
>>> Best,
>>> Flavio
>>>
>>
>>
Re: Dataset output callback
Posted by Stephan Ewen <se...@apache.org>.
If you just want to do something after the job finishes, just put the
finalization code after the call to "execute()":
env.execute();
myCustomOutputFinalization();
If you want each parallel output to do something after it finished, then
override the "close()" method of the OutputFormat.
To have a finalization call on the JobManager, make sure the OutputFormat
implements "FinalizeOnMaster" and override that interface's method.
Does any of that work for you?
Am 27.05.2015 08:43 schrieb "Flavio Pompermaier" <po...@okkam.it>:
> Any insight about this?
>
> On Tue, May 26, 2015 at 5:58 PM, Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> Hi to all,
>>
>> In my program I'd like to infer from a mysql table the list of directory
>> I have to output on hdfs (status=0).
>> Once my job finish to clean each directory and update the status value of
>> my sql table.
>> How can I do that in Flink? Is there any callback on the dataset.output()
>> finish?
>>
>> Best,
>> Flavio
>>
>
>
Re: Dataset output callback
Posted by Flavio Pompermaier <po...@okkam.it>.
Any insight about this?
On Tue, May 26, 2015 at 5:58 PM, Flavio Pompermaier <po...@okkam.it>
wrote:
> Hi to all,
>
> In my program I'd like to infer from a mysql table the list of directory I
> have to output on hdfs (status=0).
> Once my job finish to clean each directory and update the status value of
> my sql table.
> How can I do that in Flink? Is there any callback on the dataset.output()
> finish?
>
> Best,
> Flavio
>