You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Robert Metzger <rm...@apache.org> on 2020/10/27 08:48:17 UTC
Re: adding core-site xml to flink1.11
Hi Shachar,
Why do you want to use the core-site.xml to configure the file system?
Since we are adding the file systems as plugins, their initialization is
customized. It might be the case that we are intentionally ignoring xml
configurations from the classpath.
You can configure the filesystem in the flink-conf.yaml file.
On Sun, Oct 25, 2020 at 7:56 AM Shachar Carmeli <ca...@gmail.com>
wrote:
> Hi,
> I'm trying to define filesystem to flink 1.11 using core-site.xml
> I tried adding in the flink-conf.yaml env.hadoop.conf.dir and I see it is
> added to the classpath
> also adding environment variable HADOOP_CONF_DIR didn't help
>
> The flink 1.11.2 is running on docker using kubernetes
>
> I added hadoop using plugin as mentioned in
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins
>
> when configure the parameters manually I can connect to the local s3a
> server
> So it looks like the flink is not reading the core-site.xml file
>
> please advise
>
> Thanks,
> Shachar
>
Re: adding core-site xml to flink1.11
Posted by Shachar Carmeli <ca...@gmail.com>.
10x
On 2020/10/27 10:42:40, Robert Metzger <rm...@apache.org> wrote:
> Hi,
>
> it seems that this is what you have to do for now. However, I see that it
> would be nice if Flink would allow reading from multiple configuration
> files, so that you can have a "common configuration" and a "per cluster"
> configuration.
>
> I filed a JIRA ticket for a feature request:
> https://issues.apache.org/jira/browse/FLINK-19828
>
>
> On Tue, Oct 27, 2020 at 10:54 AM Shachar Carmeli <ca...@gmail.com>
> wrote:
>
> > Hi,
> > Thank you for your reply,
> > WE are deploying on kubernetes and the xml is part of the common config
> > map to all flink jobs we have(or at least was for previous versions)
> >
> > This means that we need to duplicate the configuration in the
> > flink-conf.yaml for each job
> > instead of having a common configmap
> >
> > Thanks,
> > Shachar
> >
> > On 2020/10/27 08:48:17, Robert Metzger <rm...@apache.org> wrote:
> > > Hi Shachar,
> > >
> > > Why do you want to use the core-site.xml to configure the file system?
> > >
> > > Since we are adding the file systems as plugins, their initialization is
> > > customized. It might be the case that we are intentionally ignoring xml
> > > configurations from the classpath.
> > > You can configure the filesystem in the flink-conf.yaml file.
> > >
> > >
> > > On Sun, Oct 25, 2020 at 7:56 AM Shachar Carmeli <ca...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > I'm trying to define filesystem to flink 1.11 using core-site.xml
> > > > I tried adding in the flink-conf.yaml env.hadoop.conf.dir and I see it
> > is
> > > > added to the classpath
> > > > also adding environment variable HADOOP_CONF_DIR didn't help
> > > >
> > > > The flink 1.11.2 is running on docker using kubernetes
> > > >
> > > > I added hadoop using plugin as mentioned in
> > > >
> > https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins
> > > >
> > > > when configure the parameters manually I can connect to the local s3a
> > > > server
> > > > So it looks like the flink is not reading the core-site.xml file
> > > >
> > > > please advise
> > > >
> > > > Thanks,
> > > > Shachar
> > > >
> > >
> >
>
Re: adding core-site xml to flink1.11
Posted by Robert Metzger <rm...@apache.org>.
Hi,
it seems that this is what you have to do for now. However, I see that it
would be nice if Flink would allow reading from multiple configuration
files, so that you can have a "common configuration" and a "per cluster"
configuration.
I filed a JIRA ticket for a feature request:
https://issues.apache.org/jira/browse/FLINK-19828
On Tue, Oct 27, 2020 at 10:54 AM Shachar Carmeli <ca...@gmail.com>
wrote:
> Hi,
> Thank you for your reply,
> WE are deploying on kubernetes and the xml is part of the common config
> map to all flink jobs we have(or at least was for previous versions)
>
> This means that we need to duplicate the configuration in the
> flink-conf.yaml for each job
> instead of having a common configmap
>
> Thanks,
> Shachar
>
> On 2020/10/27 08:48:17, Robert Metzger <rm...@apache.org> wrote:
> > Hi Shachar,
> >
> > Why do you want to use the core-site.xml to configure the file system?
> >
> > Since we are adding the file systems as plugins, their initialization is
> > customized. It might be the case that we are intentionally ignoring xml
> > configurations from the classpath.
> > You can configure the filesystem in the flink-conf.yaml file.
> >
> >
> > On Sun, Oct 25, 2020 at 7:56 AM Shachar Carmeli <ca...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I'm trying to define filesystem to flink 1.11 using core-site.xml
> > > I tried adding in the flink-conf.yaml env.hadoop.conf.dir and I see it
> is
> > > added to the classpath
> > > also adding environment variable HADOOP_CONF_DIR didn't help
> > >
> > > The flink 1.11.2 is running on docker using kubernetes
> > >
> > > I added hadoop using plugin as mentioned in
> > >
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins
> > >
> > > when configure the parameters manually I can connect to the local s3a
> > > server
> > > So it looks like the flink is not reading the core-site.xml file
> > >
> > > please advise
> > >
> > > Thanks,
> > > Shachar
> > >
> >
>
Re: adding core-site xml to flink1.11
Posted by Shachar Carmeli <ca...@gmail.com>.
Hi,
Thank you for your reply,
WE are deploying on kubernetes and the xml is part of the common config map to all flink jobs we have(or at least was for previous versions)
This means that we need to duplicate the configuration in the flink-conf.yaml for each job
instead of having a common configmap
Thanks,
Shachar
On 2020/10/27 08:48:17, Robert Metzger <rm...@apache.org> wrote:
> Hi Shachar,
>
> Why do you want to use the core-site.xml to configure the file system?
>
> Since we are adding the file systems as plugins, their initialization is
> customized. It might be the case that we are intentionally ignoring xml
> configurations from the classpath.
> You can configure the filesystem in the flink-conf.yaml file.
>
>
> On Sun, Oct 25, 2020 at 7:56 AM Shachar Carmeli <ca...@gmail.com>
> wrote:
>
> > Hi,
> > I'm trying to define filesystem to flink 1.11 using core-site.xml
> > I tried adding in the flink-conf.yaml env.hadoop.conf.dir and I see it is
> > added to the classpath
> > also adding environment variable HADOOP_CONF_DIR didn't help
> >
> > The flink 1.11.2 is running on docker using kubernetes
> >
> > I added hadoop using plugin as mentioned in
> > https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins
> >
> > when configure the parameters manually I can connect to the local s3a
> > server
> > So it looks like the flink is not reading the core-site.xml file
> >
> > please advise
> >
> > Thanks,
> > Shachar
> >
>