You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Mauro Cortellazzi <ma...@radicalbit.io> on 2017/05/17 10:31:54 UTC

Question about jobmanager.web.upload.dir

Hi Flink comunity,

is there a particular reason to delete the jobmanager.web.upload.dir 
content when the JobManager restart? Could it be interesting have the 
jar previously uploaded when JM restart? Could the jars be saved to HDFS 
for HA mode?

Thank you for your support.

Mauro

Re: Question about jobmanager.web.upload.dir

Posted by Mauro Cortellazzi <ma...@radicalbit.io>.

Thank you Chesnay and Timo for your answers.


Il 17/05/2017 16:49, Chesnay Schepler ha scritto:
> I don't know why we delete it either, but my guess is that at one 
> point this was a temporary directory that we properly cleaned up, and 
> later allowed it to be configurable.
>
> Currently, this directory must be in the local file system. We could 
> change it to also allow non-local paths, which should be fairly easy 
> to do. Add a condition that the directory is only cleaned up when it 
> wasn't explicitly configured and we should've covered everything.
>
> In regards to the HistoryServer, i mean of course could we include the 
> user-jar, but I'm wondering about the user-cases for it.
> It's not like you could "just download it and give it to a 
> JobManager"; you would be missing any additional libraries put under 
> /lib, as well as the original configuration of the cluster. It would 
> also require extra steps (and tools) to identify which dependencies 
> are required.
>
> And there are downsides to this, like a significant increase in the 
> size of the job archives (along with a duplication of jars for every 
> job submission), and the archiving also becomes more complex.
>
> On 17.05.2017 15:52, Timo Walther wrote:
>> Hey Mauro,
>>
>> I'm not aware of any reason for that. I loop in Chesnay, maybe he 
>> knows why. @Chesnay wouldn't it be helpful to also archive the jars 
>> using the HistoryServer?
>>
>> Timo
>>
>>
>> Am 17.05.17 um 12:31 schrieb Mauro Cortellazzi:
>>> Hi Flink comunity,
>>>
>>> is there a particular reason to delete the jobmanager.web.upload.dir 
>>> content when the JobManager restart? Could it be interesting have 
>>> the jar previously uploaded when JM restart? Could the jars be saved 
>>> to HDFS for HA mode?
>>>
>>> Thank you for your support.
>>>
>>> Mauro
>>
>>
>>
>

Re: Question about jobmanager.web.upload.dir

Posted by Chesnay Schepler <ch...@apache.org>.

I don't know why we delete it either, but my guess is that at one point 
this was a temporary directory that we properly cleaned up, and later 
allowed it to be configurable.

Currently, this directory must be in the local file system. We could 
change it to also allow non-local paths, which should be fairly easy to 
do. Add a condition that the directory is only cleaned up when it wasn't 
explicitly configured and we should've covered everything.

In regards to the HistoryServer, i mean of course could we include the 
user-jar, but I'm wondering about the user-cases for it.
It's not like you could "just download it and give it to a JobManager"; 
you would be missing any additional libraries put under /lib, as well as 
the original configuration of the cluster. It would also require extra 
steps (and tools) to identify which dependencies are required.

And there are downsides to this, like a significant increase in the size 
of the job archives (along with a duplication of jars for every job 
submission), and the archiving also becomes more complex.

On 17.05.2017 15:52, Timo Walther wrote:
> Hey Mauro,
>
> I'm not aware of any reason for that. I loop in Chesnay, maybe he 
> knows why. @Chesnay wouldn't it be helpful to also archive the jars 
> using the HistoryServer?
>
> Timo
>
>
> Am 17.05.17 um 12:31 schrieb Mauro Cortellazzi:
>> Hi Flink comunity,
>>
>> is there a particular reason to delete the jobmanager.web.upload.dir 
>> content when the JobManager restart? Could it be interesting have the 
>> jar previously uploaded when JM restart? Could the jars be saved to 
>> HDFS for HA mode?
>>
>> Thank you for your support.
>>
>> Mauro
>
>
>

Re: Question about jobmanager.web.upload.dir

Posted by Timo Walther <tw...@apache.org>.

Hey Mauro,

I'm not aware of any reason for that. I loop in Chesnay, maybe he knows 
why. @Chesnay wouldn't it be helpful to also archive the jars using the 
HistoryServer?

Timo


Am 17.05.17 um 12:31 schrieb Mauro Cortellazzi:
> Hi Flink comunity,
>
> is there a particular reason to delete the jobmanager.web.upload.dir 
> content when the JobManager restart? Could it be interesting have the 
> jar previously uploaded when JM restart? Could the jars be saved to 
> HDFS for HA mode?
>
> Thank you for your support.
>
> Mauro