You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by "zjfplayer@hotmail.com" <zj...@hotmail.com> on 2019/12/23 02:09:37 UTC
using thin jar to replace fat jar on yarn cluster mode
Hi,
Does flink on yarn support using thin jar to replace fat jar?
I don't want the jar of each flink task to have hundreds of MB. I want to put all the dependent packages in a single directory,and then the size of each flink task jar will be tens of KB.
________________________________
zjfplayer@hotmail.com
Re: Re: using thin jar to replace fat jar on yarn cluster mode
Posted by "zjfplayer@hotmail.com" <zj...@hotmail.com>.
When using yarn originally, we found a problem. The jars on yarn will take precedence over the jars on the specified classpath, and sometimes yarn is shared, and the jars in the lib directory of yarn cannot be modified. When Flink runs on a yarn cluster, is the jar on the classpath of flink itself higher than the jar on yarn?
________________________________
zjfplayer@hotmail.com
From: tangjunliang@huitongjy.com<ma...@huitongjy.com>
Date: 2019-12-23 10:34
To: user-zh<ma...@flink.apache.org>
Subject: Re: using thin jar to replace fat jar on yarn cluster mode
Specify classpath
tangjunliang@huitongjy.com
From: zjfplayer@hotmail.com
Date: 2019-12-23 10:09
To: user; user-zh
Subject: using thin jar to replace fat jar on yarn cluster mode
Hi,
Does flink on yarn support using thin jar to replace fat jar?
I don't want the jar of each flink task to have hundreds of MB. I want to put all the dependent packages in a single directory,and then the size of each flink task jar will be tens of KB.
________________________________
zjfplayer@hotmail.com
Re: using thin jar to replace fat jar on yarn cluster mode
Posted by "tangjunliang@huitongjy.com" <ta...@huitongjy.com>.
Specify classpath
tangjunliang@huitongjy.com
From: zjfplayer@hotmail.com
Date: 2019-12-23 10:09
To: user; user-zh
Subject: using thin jar to replace fat jar on yarn cluster mode
Hi,
Does flink on yarn support using thin jar to replace fat jar?
I don't want the jar of each flink task to have hundreds of MB. I want to put all the dependent packages in a single directory,and then the size of each flink task jar will be tens of KB.
________________________________
zjfplayer@hotmail.com
Re: using thin jar to replace fat jar on yarn cluster mode
Posted by Jingsong Li <ji...@gmail.com>.
Hi zjfplayer,
First, your user fat jar should not contains any flink dependents, them
should be "provided".
If your goal is to reduce the deployment time of task, and you are
currently using session mode, you can consider putting jars directly under
lib.
Best,
Jingsong Lee
On Mon, Dec 23, 2019 at 3:24 PM Rui Li <li...@gmail.com> wrote:
> Hi,
>
> I think you can try specifying dependent jars with the -C option[1] when
> you submit the job, and see if that meets your needs.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html#usage
>
> On Mon, Dec 23, 2019 at 10:09 AM zjfplayer@hotmail.com <
> zjfplayer@hotmail.com> wrote:
>
>> Hi,
>> Does flink on yarn support using thin jar to replace fat jar?
>> I don't want the jar of each flink task to have hundreds of MB. I
>> want to put all the dependent packages in a single directory,and then
>> the size of each flink task jar will be tens of KB.
>>
>> ------------------------------
>> zjfplayer@hotmail.com
>>
>
>
> --
> Best regards!
> Rui Li
>
--
Best, Jingsong Lee
Re: using thin jar to replace fat jar on yarn cluster mode
Posted by Jingsong Li <ji...@gmail.com>.
Hi zjfplayer,
First, your user fat jar should not contains any flink dependents, them
should be "provided".
If your goal is to reduce the deployment time of task, and you are
currently using session mode, you can consider putting jars directly under
lib.
Best,
Jingsong Lee
On Mon, Dec 23, 2019 at 3:24 PM Rui Li <li...@gmail.com> wrote:
> Hi,
>
> I think you can try specifying dependent jars with the -C option[1] when
> you submit the job, and see if that meets your needs.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html#usage
>
> On Mon, Dec 23, 2019 at 10:09 AM zjfplayer@hotmail.com <
> zjfplayer@hotmail.com> wrote:
>
>> Hi,
>> Does flink on yarn support using thin jar to replace fat jar?
>> I don't want the jar of each flink task to have hundreds of MB. I
>> want to put all the dependent packages in a single directory,and then
>> the size of each flink task jar will be tens of KB.
>>
>> ------------------------------
>> zjfplayer@hotmail.com
>>
>
>
> --
> Best regards!
> Rui Li
>
--
Best, Jingsong Lee
Re: using thin jar to replace fat jar on yarn cluster mode
Posted by Rui Li <li...@gmail.com>.
Hi,
I think you can try specifying dependent jars with the -C option[1] when
you submit the job, and see if that meets your needs.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html#usage
On Mon, Dec 23, 2019 at 10:09 AM zjfplayer@hotmail.com <
zjfplayer@hotmail.com> wrote:
> Hi,
> Does flink on yarn support using thin jar to replace fat jar?
> I don't want the jar of each flink task to have hundreds of MB. I
> want to put all the dependent packages in a single directory,and then the
> size of each flink task jar will be tens of KB.
>
> ------------------------------
> zjfplayer@hotmail.com
>
--
Best regards!
Rui Li
Re: using thin jar to replace fat jar on yarn cluster mode
Posted by Rui Li <li...@gmail.com>.
Hi,
I think you can try specifying dependent jars with the -C option[1] when
you submit the job, and see if that meets your needs.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html#usage
On Mon, Dec 23, 2019 at 10:09 AM zjfplayer@hotmail.com <
zjfplayer@hotmail.com> wrote:
> Hi,
> Does flink on yarn support using thin jar to replace fat jar?
> I don't want the jar of each flink task to have hundreds of MB. I
> want to put all the dependent packages in a single directory,and then the
> size of each flink task jar will be tens of KB.
>
> ------------------------------
> zjfplayer@hotmail.com
>
--
Best regards!
Rui Li