You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by C DINESH <di...@gmail.com> on 2020/08/06 11:08:34 UTC

Hadoop_Compatability

Hi All,

From 1.9 version there is no *flink-shaded-hadoop2 dependency. To use
Hadoop APIS like *IntWritable ,  LongWritable. What are the dependencies we
need to add to use these APIs.

I tried searching in google. Not able to understand the solution. Please
guide me.

Thanks in Advance.
Dinesh.

Re: Hadoop_Compatability

Posted by C DINESH <di...@gmail.com>.

Thanks for the response Chesnay,

i will try to understand it. If I have doubts I will get back.

Thanks & Regards,
Dinesh.

On Thu, Aug 6, 2020 at 5:11 PM Chesnay Schepler <ch...@apache.org> wrote:

> We still offer a flink-shaded-hadoop-2 artifact that you can find on the
> download page:
> https://flink.apache.org/downloads.html#additional-components
> In 1.9 we changed the artifact name.
>
> Note that we will not release newer versions of this dependency.
>
> As for providing Hadoop class, there is some guidance in the documentation
> <https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html#providing-hadoop-classes>
> .
> If this does not work fr your, and you want to put the required
> dependencies into the lib/ directory, then you will have to look through
> your dependency tree to determine which Hadoop dependencies are required.
> The Flink hadoop-compatibility dependency requires hadoop-common and
> hadoop-mapreduce-client-core, and whatever transitive dependencies these
> have.
>
> On 06/08/2020 13:08, C DINESH wrote:
>
> Hi All,
>
> From 1.9 version there is no *flink-shaded-hadoop2 dependency. To use
> Hadoop APIS like *IntWritable ,  LongWritable. What are the dependencies
> we need to add to use these APIs.
>
> I tried searching in google. Not able to understand the solution. Please
> guide me.
>
> Thanks in Advance.
> Dinesh.
>
>
>

Re: Hadoop_Compatability

Posted by Chesnay Schepler <ch...@apache.org>.

We still offer a flink-shaded-hadoop-2 artifact that you can find on the 
download page: https://flink.apache.org/downloads.html#additional-components
In 1.9 we changed the artifact name.

Note that we will not release newer versions of this dependency.

As for providing Hadoop class, there is some guidance in the 
documentation 
<https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html#providing-hadoop-classes>.
If this does not work fr your, and you want to put the required 
dependencies into the lib/ directory, then you will have to look through 
your dependency tree to determine which Hadoop dependencies are required.
The Flink hadoop-compatibility dependency requires hadoop-common and 
hadoop-mapreduce-client-core, and whatever transitive dependencies these 
have.

On 06/08/2020 13:08, C DINESH wrote:
> Hi All,
>
> From 1.9 version there is no *flink-shaded-hadoop2dependency. To use 
> Hadoop APIS like *IntWritable , LongWritable. What are the 
> dependencies we need to add to use these APIs.
>
> I tried searching in google. Not able to understand the solution. 
> Please guide me.
>
> Thanks in Advance.
> Dinesh.