You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Jacky Lau <li...@gmail.com> on 2022/10/27 04:58:48 UTC
[DISCUSS ] add --jars to support users dependencies jars.
Hi guys:
I'd like to initiate a discussion about adding command-line arguments to
support user-dependent jar packages.
Currently flink supports user's main jars through -jarfile or without
setting this , the flink client will treat the first argument after that as
the user master jar package when it encounters the first command line
argument that cannot be parsed. but it is not possible for platform users
to create fat jars to package all their dependencies into the final jar
package. In the meantime, the configuration pipeline.jars is currently
exposed, and this value is overridden by command-line arguments such as
-jarfile.
And If the user is using both the command-line argument and the
pipeline.jars argument, which can make the user werild and confused. In
addition, we should specify the priority "command line parameter > -D
dynamic parameter > flink-conf.yml configuration file parameter" in docs
Re: [DISCUSS ] add --jars to support users dependencies jars.
Posted by Yang Wang <da...@gmail.com>.
Thanks Jacky Lau for starting this discussion.
I understand that you are trying to find a convenient way to specify
dependency jars along with user jar. However,
let's try to narrow down by differentiating deployment modes.
# Standalone mode
No matter you are using the standalone mode on virtual machine, or in a
Kubernetes cluster,
it is not very difficult to preparing user jar and all the dependencies
under the $FLINK_HOME/usrlib directory.
After then, they will be loaded by user classloader automatically.
# Yarn
We already have "--ship/-Dyarn.ship-files" to ship the dependency jars.
# Native K8s
Currently, only the local user jar in the image could be supported. And
users could not specify dependency jars.
A feasible solution is using the init-container(via pod template[1]) to
download the user jar and dependencies and then mount to usrlib directory.
All in all, I trying to get you point about why do we need the "--jars" to
specify the dependency jars. And which deployment mode it will support?
Best,
Yang
[1].
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/native_kubernetes/#pod-template
Martijn Visser <ma...@apache.org> 于2022年10月27日周四 14:49写道：
> Hi Jacky Lau,
>
> Since you've sent the email to multiple mailing lists, I've decided to
> reply to the one that you've sent to both the Dev and User ML.
>
> > but it is not possible for platform users to create fat jars to package
> all their dependencies into the final jar package
>
> Can you elaborate on why that's not possible?
>
> Best regards,
>
> Martijn
>
> On Thu, Oct 27, 2022 at 6:59 AM Jacky Lau <li...@gmail.com> wrote:
>
> > Hi guys:
> >
> > I'd like to initiate a discussion about adding command-line arguments to
> > support user-dependent jar packages.
> >
> > Currently flink supports user's main jars through -jarfile or without
> > setting this , the flink client will treat the first argument after that
> as
> > the user master jar package when it encounters the first command line
> > argument that cannot be parsed. but it is not possible for platform users
> > to create fat jars to package all their dependencies into the final jar
> > package. In the meantime, the configuration pipeline.jars is currently
> > exposed, and this value is overridden by command-line arguments such as
> > -jarfile.
> >
> > And If the user is using both the command-line argument and the
> > pipeline.jars argument, which can make the user werild and confused. In
> > addition, we should specify the priority "command line parameter > -D
> > dynamic parameter > flink-conf.yml configuration file parameter" in docs
> >
>
Re: [DISCUSS ] add --jars to support users dependencies jars.
Posted by Yang Wang <da...@gmail.com>.
Thanks Jacky Lau for starting this discussion.
I understand that you are trying to find a convenient way to specify
dependency jars along with user jar. However,
let's try to narrow down by differentiating deployment modes.
# Standalone mode
No matter you are using the standalone mode on virtual machine, or in a
Kubernetes cluster,
it is not very difficult to preparing user jar and all the dependencies
under the $FLINK_HOME/usrlib directory.
After then, they will be loaded by user classloader automatically.
# Yarn
We already have "--ship/-Dyarn.ship-files" to ship the dependency jars.
# Native K8s
Currently, only the local user jar in the image could be supported. And
users could not specify dependency jars.
A feasible solution is using the init-container(via pod template[1]) to
download the user jar and dependencies and then mount to usrlib directory.
All in all, I trying to get you point about why do we need the "--jars" to
specify the dependency jars. And which deployment mode it will support?
Best,
Yang
[1].
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/native_kubernetes/#pod-template
Martijn Visser <ma...@apache.org> 于2022年10月27日周四 14:49写道：
> Hi Jacky Lau,
>
> Since you've sent the email to multiple mailing lists, I've decided to
> reply to the one that you've sent to both the Dev and User ML.
>
> > but it is not possible for platform users to create fat jars to package
> all their dependencies into the final jar package
>
> Can you elaborate on why that's not possible?
>
> Best regards,
>
> Martijn
>
> On Thu, Oct 27, 2022 at 6:59 AM Jacky Lau <li...@gmail.com> wrote:
>
> > Hi guys:
> >
> > I'd like to initiate a discussion about adding command-line arguments to
> > support user-dependent jar packages.
> >
> > Currently flink supports user's main jars through -jarfile or without
> > setting this , the flink client will treat the first argument after that
> as
> > the user master jar package when it encounters the first command line
> > argument that cannot be parsed. but it is not possible for platform users
> > to create fat jars to package all their dependencies into the final jar
> > package. In the meantime, the configuration pipeline.jars is currently
> > exposed, and this value is overridden by command-line arguments such as
> > -jarfile.
> >
> > And If the user is using both the command-line argument and the
> > pipeline.jars argument, which can make the user werild and confused. In
> > addition, we should specify the priority "command line parameter > -D
> > dynamic parameter > flink-conf.yml configuration file parameter" in docs
> >
>
Re: [DISCUSS ] add --jars to support users dependencies jars.
Posted by Martijn Visser <ma...@apache.org>.
Hi Jacky Lau,
Since you've sent the email to multiple mailing lists, I've decided to
reply to the one that you've sent to both the Dev and User ML.
> but it is not possible for platform users to create fat jars to package
all their dependencies into the final jar package
Can you elaborate on why that's not possible?
Best regards,
Martijn
On Thu, Oct 27, 2022 at 6:59 AM Jacky Lau <li...@gmail.com> wrote:
> Hi guys:
>
> I'd like to initiate a discussion about adding command-line arguments to
> support user-dependent jar packages.
>
> Currently flink supports user's main jars through -jarfile or without
> setting this , the flink client will treat the first argument after that as
> the user master jar package when it encounters the first command line
> argument that cannot be parsed. but it is not possible for platform users
> to create fat jars to package all their dependencies into the final jar
> package. In the meantime, the configuration pipeline.jars is currently
> exposed, and this value is overridden by command-line arguments such as
> -jarfile.
>
> And If the user is using both the command-line argument and the
> pipeline.jars argument, which can make the user werild and confused. In
> addition, we should specify the priority "command line parameter > -D
> dynamic parameter > flink-conf.yml configuration file parameter" in docs
>
Re: [DISCUSS ] add --jars to support users dependencies jars.
Posted by Martijn Visser <ma...@apache.org>.
Hi Jacky Lau,
Since you've sent the email to multiple mailing lists, I've decided to
reply to the one that you've sent to both the Dev and User ML.
> but it is not possible for platform users to create fat jars to package
all their dependencies into the final jar package
Can you elaborate on why that's not possible?
Best regards,
Martijn
On Thu, Oct 27, 2022 at 6:59 AM Jacky Lau <li...@gmail.com> wrote:
> Hi guys:
>
> I'd like to initiate a discussion about adding command-line arguments to
> support user-dependent jar packages.
>
> Currently flink supports user's main jars through -jarfile or without
> setting this , the flink client will treat the first argument after that as
> the user master jar package when it encounters the first command line
> argument that cannot be parsed. but it is not possible for platform users
> to create fat jars to package all their dependencies into the final jar
> package. In the meantime, the configuration pipeline.jars is currently
> exposed, and this value is overridden by command-line arguments such as
> -jarfile.
>
> And If the user is using both the command-line argument and the
> pipeline.jars argument, which can make the user werild and confused. In
> addition, we should specify the priority "command line parameter > -D
> dynamic parameter > flink-conf.yml configuration file parameter" in docs
>