You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "yinhua.dai" <yi...@outlook.com> on 2018/12/28 03:52:43 UTC
Flink SQL client always cost 2 minutes to submit job to a local
cluster
I am using Flink 1.6.1, I tried to use flink sql client with some own jars
with --jar and --library.
It can work to execute sql query, however it always cause around 2 minutes
to submit the job the local cluster, but when I copy my jar to flink lib,
and remove --jar and --library parameter, it can submit the job immediately.
I debugged and found the 2 minutes is cost by the RestClusterClient to send
the request with the jar as payload to the flink cluster.
I don't know why it uses 2 minutes to upload the package? Is there a way to
work around it?
Thanks.
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Flink SQL client always cost 2 minutes to submit job to a local cluster
Posted by Hequn Cheng <ch...@gmail.com>.
Hi yinhua,
Could you help to reproduce the problem? I can help to figure out the root
cause.
Best, Hequn
On Fri, Jan 4, 2019 at 11:37 AM yinhua.dai <yi...@outlook.com> wrote:
> Hi Fabian,
>
> It's the submission of the jar file cost too long time.
> And yes Hequn and your suggestion is working, but just curious why a 100M
> jar files causes so long time to submit, is it related with some upload
> parameter settings of the web layer?
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Flink SQL client always cost 2 minutes to submit job to a local
cluster
Posted by "yinhua.dai" <yi...@outlook.com>.
Hi Fabian,
It's the submission of the jar file cost too long time.
And yes Hequn and your suggestion is working, but just curious why a 100M
jar files causes so long time to submit, is it related with some upload
parameter settings of the web layer?
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Flink SQL client always cost 2 minutes to submit job to a local cluster
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
You can try to build a JAR file with all runtime dependencies of Flink SQL
(Calcite, Janino, +transitive dependencies), add it to the lib folder, and
exclude the dependencies from the JAR file that is sent to the cluster when
you submit a job.
It would also be good to figure out what takes so long.
Is it actually the submission of the jar file (in that case Hequn's and my
suggestions would help) or the time to plan a query?
Best, Fabian
Am So., 30. Dez. 2018 um 22:06 Uhr schrieb yinhua.dai <
yinhua.2018@outlook.com>:
> I have to do that for now, however I have to find another way because the
> jar
> some times get update and the flink cluster will be remotely in future.
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Flink SQL client always cost 2 minutes to submit job to a local
cluster
Posted by "yinhua.dai" <yi...@outlook.com>.
I have to do that for now, however I have to find another way because the jar
some times get update and the flink cluster will be remotely in future.
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Flink SQL client always cost 2 minutes to submit job to a local cluster
Posted by Hequn Cheng <ch...@gmail.com>.
Hi, yinhua
Thanks for looking into the problem. I'm not familiar with the code of
these part. As a workaround, you can put your jars into the flink lib
folder or add your jars into the classpath. Hope this helps.
Best, Hequn
On Fri, Dec 28, 2018 at 11:52 AM yinhua.dai <yi...@outlook.com> wrote:
> I am using Flink 1.6.1, I tried to use flink sql client with some own jars
> with --jar and --library.
> It can work to execute sql query, however it always cause around 2 minutes
> to submit the job the local cluster, but when I copy my jar to flink lib,
> and remove --jar and --library parameter, it can submit the job
> immediately.
>
> I debugged and found the 2 minutes is cost by the RestClusterClient to send
> the request with the jar as payload to the flink cluster.
>
> I don't know why it uses 2 minutes to upload the package? Is there a way to
> work around it?
> Thanks.
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>