External classpath

[Vishwas Siravara <vs...@gmail.com>]

Hi guys,
I m very close to deploying my application in production so I am trying to
externalize some of the config files which has to be available on the
classpath when I run my application via flink command line interface.

From the flink doc I can add to class path by

-C,--classpath <url>                 Adds a URL to each user code
                                          classloader  on all nodes in the
                                          cluster. The paths must specify a
                                          protocol (e.g. file://) and be
                                          accessible on all nodes (e.g. by means
                                          of a NFS share). You can use this
                                          option multiple times for specifying
                                          more than one URL. The protocol must
                                          be supported by the {@link
                                          java.net.URLClassLoader}.

So my job has 8 task managers( 8 different nodes) with 8 slots in each .
Does my external class path have to be on NFS share ? Can I not have the
config directory on each machine in the same location ? For instance on
Node 1 the config files are in  the directory is
/home/was/classpathconfig/  and
the same on every node. Does it have to be on an NFS ?
My command looks like this flink run -C file://home/was/classpathconfig/ -c
com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Also I tried to put my files in s3 and tried to run flink run -C s3://
flink.dev/config -c com.visa.flink.cli.Main
flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Bad syntax for classpath: s3://flink.dev/config


s3 does support URLClassLoader but I get the error saying bad syntax.


Please let me know your thoughts. Thanks a lot to this community , I
was able to write my code in a week.


Thanks,

Vishwas

Re: External classpath

[Biao Liu <mm...@gmail.com>]

Hi Vishwas,

> Does my external class path have to be on NFS share ? Can I not have the
config directory on each machine in the same location ?

It can be local files on each machine.

> -C file://home/was/classpathconfig/

I guess the problem is your URI format of local file is not correct. Try to
use "-C file:///home/was/classpathconfig/" instead.
BTW, if it's a directory, it must end with '/' [1].

> Also I tried to put my files in s3 and tried to run flink run -C s3://
flink.dev/config -c com.visa.flink.cli.Main
flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Currently it doesn't support s3 or other distributed file system as the
parameter of -C.

1. https://docs.oracle.com/javase/8/docs/api/java/net/URLClassLoader.html

Thanks,
Biao /'bɪ.aʊ/



On Thu, 15 Aug 2019 at 07:51, Vishwas Siravara <vs...@gmail.com> wrote:

> Hi guys,
> I m very close to deploying my application in production so I am trying to
> externalize some of the config files which has to be available on the
> classpath when I run my application via flink command line interface.
>
> From the flink doc I can add to class path by
>
> -C,--classpath <url>                 Adds a URL to each user code
>                                           classloader  on all nodes in the
>                                           cluster. The paths must specify a
>                                           protocol (e.g. file://) and be
>                                           accessible on all nodes (e.g. by means
>                                           of a NFS share). You can use this
>                                           option multiple times for specifying
>                                           more than one URL. The protocol must
>                                           be supported by the {@link
>                                           java.net.URLClassLoader}.
>
> So my job has 8 task managers( 8 different nodes) with 8 slots in each .
> Does my external class path have to be on NFS share ? Can I not have the
> config directory on each machine in the same location ? For instance on
> Node 1 the config files are in  the directory is
> /home/was/classpathconfig/  and the same on every node. Does it have to
> be on an NFS ?
> My command looks like this flink run -C file://home/was/classpathconfig/
> -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid
>
> Also I tried to put my files in s3 and tried to run flink run -C s3://
> flink.dev/config -c com.visa.flink.cli.Main
> flink-job-assembly-0.1-SNAPSHOT.jar flink druid
>
> Bad syntax for classpath: s3://flink.dev/config
>
>
> s3 does support URLClassLoader but I get the error saying bad syntax.
>
>
> Please let me know your thoughts. Thanks a lot to this community , I was able to write my code in a week.
>
>
> Thanks,
>
> Vishwas
>
>