You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexander Alexandrov <al...@gmail.com> on 2015/01/15 01:00:20 UTC
Customizing the classpath for TM and JM
Hi there,
is there a canonical / suggested way to customize the classpath of the
Flink JM and TM processes?
At the moment I hardcoded my way around this by manually changing the
following lines like that:
# in taskmanager.sh
$JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
# in jobmanagre.sh
$JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
Perhaps there is a variable that I can set in conf/flink.yaml that will do
the same trick.
Re: Customizing the classpath for TM and JM
Posted by Ufuk Celebi <uc...@apache.org>.
On Thursday, January 15, 2015, Alexander Alexandrov <
alexander.s.alexandrov@gmail.com> wrote:
> OK, let's be conservative and not blow up the list of configuration values
> then.
>
Yes :)
> Is there a default point where I should also document these settings?
>
In the setup section of the docs. There is an overview page of all config
parameters.
> 2015-01-15 15:02 GMT+01:00 Ufuk Celebi <uce@apache.org
> <javascript:_e(%7B%7D,'cvml','uce@apache.org');>>:
>
>> "env.java.classpath" could be used for global class path settings (both
>> JM and TM) and the other ones for JM and TM respectively. I don't know if
>> this is really necessary though. Because no one else asked for it, I think
>> it's fine to go with what suits your use case.
>>
>> – Ufuk
>>
>> On 15 Jan 2015, at 14:25, Alexander Alexandrov <
>> alexander.s.alexandrov@gmail.com
>> <javascript:_e(%7B%7D,'cvml','alexander.s.alexandrov@gmail.com');>>
>> wrote:
>>
>> > Sorry, I meant between the three. In particular, where will
>> env.java.classpath propagate?
>> >
>> > 2015-01-15 14:24 GMT+01:00 Alexander Alexandrov <
>> alexander.s.alexandrov@gmail.com
>> <javascript:_e(%7B%7D,'cvml','alexander.s.alexandrov@gmail.com');>>:
>> > No worries, this is the first reply that I get in this thread.
>> >
>> > What would be the difference between the two? I can make a PR if we
>> agree on that.
>> >
>> > 2015-01-15 11:28 GMT+01:00 Ufuk Celebi <uce@apache.org
>> <javascript:_e(%7B%7D,'cvml','uce@apache.org');>>:
>> > Hey Alexander,
>> >
>> > there is only the option "env.java.opts", which can be set in the
>> config. I think you cannot use this to have multiple "-classpath" options
>> when starting Java?
>> >
>> > It makes sense to have the option to add something to the classpath. I
>> will check this and fix it. :-) What about "taskmanager.env.java.classpath"
>> and "jobmanager.env.java.classpath" as a configuration key? And maybe an
>> additional "env.java.classpath"?
>> >
>> > PS: sorry for the incomplete mail before :D
>> >
>> > On 15 Jan 2015, at 01:00, Alexander Alexandrov <
>> alexander.s.alexandrov@gmail.com
>> <javascript:_e(%7B%7D,'cvml','alexander.s.alexandrov@gmail.com');>>
>> wrote:
>> >
>> > > Hi there,
>> > >
>> > > is there a canonical / suggested way to customize the classpath of
>> the Flink JM and TM processes?
>> > >
>> > > At the moment I hardcoded my way around this by manually changing the
>> following lines like that:
>> > >
>> > > # in taskmanager.sh
>> > > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
>> > >
>> > > # in jobmanagre.sh
>> > > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
>> > >
>> > > Perhaps there is a variable that I can set in conf/flink.yaml that
>> will do the same trick.
>> >
>> >
>> >
>>
>>
>
Re: Customizing the classpath for TM and JM
Posted by Alexander Alexandrov <al...@gmail.com>.
OK, let's be conservative and not blow up the list of configuration values
then.
Is there a default point where I should also document these settings?
2015-01-15 15:02 GMT+01:00 Ufuk Celebi <uc...@apache.org>:
> "env.java.classpath" could be used for global class path settings (both JM
> and TM) and the other ones for JM and TM respectively. I don't know if this
> is really necessary though. Because no one else asked for it, I think it's
> fine to go with what suits your use case.
>
> – Ufuk
>
> On 15 Jan 2015, at 14:25, Alexander Alexandrov <
> alexander.s.alexandrov@gmail.com> wrote:
>
> > Sorry, I meant between the three. In particular, where will
> env.java.classpath propagate?
> >
> > 2015-01-15 14:24 GMT+01:00 Alexander Alexandrov <
> alexander.s.alexandrov@gmail.com>:
> > No worries, this is the first reply that I get in this thread.
> >
> > What would be the difference between the two? I can make a PR if we
> agree on that.
> >
> > 2015-01-15 11:28 GMT+01:00 Ufuk Celebi <uc...@apache.org>:
> > Hey Alexander,
> >
> > there is only the option "env.java.opts", which can be set in the
> config. I think you cannot use this to have multiple "-classpath" options
> when starting Java?
> >
> > It makes sense to have the option to add something to the classpath. I
> will check this and fix it. :-) What about "taskmanager.env.java.classpath"
> and "jobmanager.env.java.classpath" as a configuration key? And maybe an
> additional "env.java.classpath"?
> >
> > PS: sorry for the incomplete mail before :D
> >
> > On 15 Jan 2015, at 01:00, Alexander Alexandrov <
> alexander.s.alexandrov@gmail.com> wrote:
> >
> > > Hi there,
> > >
> > > is there a canonical / suggested way to customize the classpath of the
> Flink JM and TM processes?
> > >
> > > At the moment I hardcoded my way around this by manually changing the
> following lines like that:
> > >
> > > # in taskmanager.sh
> > > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
> > >
> > > # in jobmanagre.sh
> > > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
> > >
> > > Perhaps there is a variable that I can set in conf/flink.yaml that
> will do the same trick.
> >
> >
> >
>
>
Re: Customizing the classpath for TM and JM
Posted by Ufuk Celebi <uc...@apache.org>.
"env.java.classpath" could be used for global class path settings (both JM and TM) and the other ones for JM and TM respectively. I don't know if this is really necessary though. Because no one else asked for it, I think it's fine to go with what suits your use case.
– Ufuk
On 15 Jan 2015, at 14:25, Alexander Alexandrov <al...@gmail.com> wrote:
> Sorry, I meant between the three. In particular, where will env.java.classpath propagate?
>
> 2015-01-15 14:24 GMT+01:00 Alexander Alexandrov <al...@gmail.com>:
> No worries, this is the first reply that I get in this thread.
>
> What would be the difference between the two? I can make a PR if we agree on that.
>
> 2015-01-15 11:28 GMT+01:00 Ufuk Celebi <uc...@apache.org>:
> Hey Alexander,
>
> there is only the option "env.java.opts", which can be set in the config. I think you cannot use this to have multiple "-classpath" options when starting Java?
>
> It makes sense to have the option to add something to the classpath. I will check this and fix it. :-) What about "taskmanager.env.java.classpath" and "jobmanager.env.java.classpath" as a configuration key? And maybe an additional "env.java.classpath"?
>
> PS: sorry for the incomplete mail before :D
>
> On 15 Jan 2015, at 01:00, Alexander Alexandrov <al...@gmail.com> wrote:
>
> > Hi there,
> >
> > is there a canonical / suggested way to customize the classpath of the Flink JM and TM processes?
> >
> > At the moment I hardcoded my way around this by manually changing the following lines like that:
> >
> > # in taskmanager.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
> >
> > # in jobmanagre.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
> >
> > Perhaps there is a variable that I can set in conf/flink.yaml that will do the same trick.
>
>
>
Re: Customizing the classpath for TM and JM
Posted by Alexander Alexandrov <al...@gmail.com>.
Sorry, I meant between the three. In particular, where will
env.java.classpath propagate?
2015-01-15 14:24 GMT+01:00 Alexander Alexandrov <
alexander.s.alexandrov@gmail.com>:
> No worries, this is the first reply that I get in this thread.
>
> What would be the difference between the two? I can make a PR if we agree
> on that.
>
> 2015-01-15 11:28 GMT+01:00 Ufuk Celebi <uc...@apache.org>:
>
>> Hey Alexander,
>>
>> there is only the option "env.java.opts", which can be set in the config.
>> I think you cannot use this to have multiple "-classpath" options when
>> starting Java?
>>
>> It makes sense to have the option to add something to the classpath. I
>> will check this and fix it. :-) What about "taskmanager.env.java.classpath"
>> and "jobmanager.env.java.classpath" as a configuration key? And maybe an
>> additional "env.java.classpath"?
>>
>> PS: sorry for the incomplete mail before :D
>>
>> On 15 Jan 2015, at 01:00, Alexander Alexandrov <
>> alexander.s.alexandrov@gmail.com> wrote:
>>
>> > Hi there,
>> >
>> > is there a canonical / suggested way to customize the classpath of the
>> Flink JM and TM processes?
>> >
>> > At the moment I hardcoded my way around this by manually changing the
>> following lines like that:
>> >
>> > # in taskmanager.sh
>> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
>> >
>> > # in jobmanagre.sh
>> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
>> >
>> > Perhaps there is a variable that I can set in conf/flink.yaml that will
>> do the same trick.
>>
>>
>
Re: Customizing the classpath for TM and JM
Posted by Alexander Alexandrov <al...@gmail.com>.
No worries, this is the first reply that I get in this thread.
What would be the difference between the two? I can make a PR if we agree
on that.
2015-01-15 11:28 GMT+01:00 Ufuk Celebi <uc...@apache.org>:
> Hey Alexander,
>
> there is only the option "env.java.opts", which can be set in the config.
> I think you cannot use this to have multiple "-classpath" options when
> starting Java?
>
> It makes sense to have the option to add something to the classpath. I
> will check this and fix it. :-) What about "taskmanager.env.java.classpath"
> and "jobmanager.env.java.classpath" as a configuration key? And maybe an
> additional "env.java.classpath"?
>
> PS: sorry for the incomplete mail before :D
>
> On 15 Jan 2015, at 01:00, Alexander Alexandrov <
> alexander.s.alexandrov@gmail.com> wrote:
>
> > Hi there,
> >
> > is there a canonical / suggested way to customize the classpath of the
> Flink JM and TM processes?
> >
> > At the moment I hardcoded my way around this by manually changing the
> following lines like that:
> >
> > # in taskmanager.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
> >
> > # in jobmanagre.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
> >
> > Perhaps there is a variable that I can set in conf/flink.yaml that will
> do the same trick.
>
>
Re: Customizing the classpath for TM and JM
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Alexander,
there is only the option "env.java.opts", which can be set in the config. I think you cannot use this to have multiple "-classpath" options when starting Java?
It makes sense to have the option to add something to the classpath. I will check this and fix it. :-) What about "taskmanager.env.java.classpath" and "jobmanager.env.java.classpath" as a configuration key? And maybe an additional "env.java.classpath"?
PS: sorry for the incomplete mail before :D
On 15 Jan 2015, at 01:00, Alexander Alexandrov <al...@gmail.com> wrote:
> Hi there,
>
> is there a canonical / suggested way to customize the classpath of the Flink JM and TM processes?
>
> At the moment I hardcoded my way around this by manually changing the following lines like that:
>
> # in taskmanager.sh
> $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
>
> # in jobmanagre.sh
> $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
>
> Perhaps there is a variable that I can set in conf/flink.yaml that will do the same trick.