You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "V N, Suchithra (Nokia - IN/Bangalore)" <su...@nokia.com> on 2021/06/24 09:04:52 UTC
Hadoop3 with Flink
Hello,
We are using Apache flink 1.12.3 and planning to use Hadoop 3 version. Could you please suggest how to use Hadoop 3 with flink distribution.
Regards,
Suchithra
Re: Suspected SPAM - RE: FW: Hadoop3 with Flink
Posted by Fabian Paul <fa...@data-artisans.com>.
Hi Suchithra,
I suspect you want to connect to something like HDFS from the running k8s application. Is this correct?
Yangze already gave some good hints you have to make sure that the necessary Hadoop java libraries
and configurations are part of your Flink docker images or are mounted from an external filesystem. [1]
If the Hadoop jar files are present in your image you can set the environment variable $HADOOP_CLASSPATH
to point to the jar location.
I hope that helps, please share more information about your setup if this solution does not suffice.
Best,
Fabian
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html#providing-hadoop-classes
RE: Suspected SPAM - RE: FW: Hadoop3 with Flink
Posted by "V N, Suchithra (Nokia - IN/Bangalore)" <su...@nokia.com>.
Hi,
Could you please help on this?
Regards,
Suchithra
-----Original Message-----
From: V N, Suchithra (Nokia - IN/Bangalore) <su...@nokia.com>
Sent: Wednesday, June 30, 2021 11:34 AM
To: Yangze Guo <ka...@gmail.com>; user@flink.apache.org
Subject: Suspected SPAM - RE: FW: Hadoop3 with Flink
Hi Yangze Guo,
Thanks for the reply. I am using flink in Kubernetes environment. Hence can you please suggest how to use hadoop3 with flink in k8s.
Regards,
Suchithra
-----Original Message-----
From: Yangze Guo <ka...@gmail.com>
Sent: Monday, June 28, 2021 3:16 PM
To: V N, Suchithra (Nokia - IN/Bangalore) <su...@nokia.com>
Cc: user@flink.apache.org
Subject: Re: FW: Hadoop3 with Flink
Sorry for the belated reply. In 1.12, you just need to make sure that the HADOOP_CLASSPATH environment variable is set up. For more details, please refer to [1].
[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/
Best,
Yangze Guo
On Mon, Jun 28, 2021 at 5:11 PM V N, Suchithra (Nokia - IN/Bangalore) <su...@nokia.com> wrote:
>
> Hi,
>
> Can anyone please share inputs on this?
>
>
>
> Regards,
>
> Suchithra
>
>
>
> From: V N, Suchithra (Nokia - IN/Bangalore)
> Sent: Thursday, June 24, 2021 2:35 PM
> To: user@flink.apache.org
> Subject: Hadoop3 with Flink
>
>
>
> Hello,
>
>
>
> We are using Apache flink 1.12.3 and planning to use Hadoop 3 version. Could you please suggest how to use Hadoop 3 with flink distribution.
>
>
>
> Regards,
>
> Suchithra
>
>
>
>
RE: FW: Hadoop3 with Flink
Posted by "V N, Suchithra (Nokia - IN/Bangalore)" <su...@nokia.com>.
Hi Yangze Guo,
Thanks for the reply. I am using flink in Kubernetes environment. Hence can you please suggest how to use hadoop3 with flink in k8s.
Regards,
Suchithra
-----Original Message-----
From: Yangze Guo <ka...@gmail.com>
Sent: Monday, June 28, 2021 3:16 PM
To: V N, Suchithra (Nokia - IN/Bangalore) <su...@nokia.com>
Cc: user@flink.apache.org
Subject: Re: FW: Hadoop3 with Flink
Sorry for the belated reply. In 1.12, you just need to make sure that the HADOOP_CLASSPATH environment variable is set up. For more details, please refer to [1].
[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/
Best,
Yangze Guo
On Mon, Jun 28, 2021 at 5:11 PM V N, Suchithra (Nokia - IN/Bangalore) <su...@nokia.com> wrote:
>
> Hi,
>
> Can anyone please share inputs on this?
>
>
>
> Regards,
>
> Suchithra
>
>
>
> From: V N, Suchithra (Nokia - IN/Bangalore)
> Sent: Thursday, June 24, 2021 2:35 PM
> To: user@flink.apache.org
> Subject: Hadoop3 with Flink
>
>
>
> Hello,
>
>
>
> We are using Apache flink 1.12.3 and planning to use Hadoop 3 version. Could you please suggest how to use Hadoop 3 with flink distribution.
>
>
>
> Regards,
>
> Suchithra
>
>
>
>
Re: FW: Hadoop3 with Flink
Posted by Yangze Guo <ka...@gmail.com>.
Sorry for the belated reply. In 1.12, you just need to make sure that
the HADOOP_CLASSPATH environment variable is set up. For more details,
please refer to [1].
[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/
Best,
Yangze Guo
On Mon, Jun 28, 2021 at 5:11 PM V N, Suchithra (Nokia - IN/Bangalore)
<su...@nokia.com> wrote:
>
> Hi,
>
> Can anyone please share inputs on this?
>
>
>
> Regards,
>
> Suchithra
>
>
>
> From: V N, Suchithra (Nokia - IN/Bangalore)
> Sent: Thursday, June 24, 2021 2:35 PM
> To: user@flink.apache.org
> Subject: Hadoop3 with Flink
>
>
>
> Hello,
>
>
>
> We are using Apache flink 1.12.3 and planning to use Hadoop 3 version. Could you please suggest how to use Hadoop 3 with flink distribution.
>
>
>
> Regards,
>
> Suchithra
>
>
>
>
FW: Hadoop3 with Flink
Posted by "V N, Suchithra (Nokia - IN/Bangalore)" <su...@nokia.com>.
Hi,
Can anyone please share inputs on this?
Regards,
Suchithra
From: V N, Suchithra (Nokia - IN/Bangalore)
Sent: Thursday, June 24, 2021 2:35 PM
To: user@flink.apache.org
Subject: Hadoop3 with Flink
Hello,
We are using Apache flink 1.12.3 and planning to use Hadoop 3 version. Could you please suggest how to use Hadoop 3 with flink distribution.
Regards,
Suchithra