You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Gyula Fóra <gy...@apache.org> on 2016/01/22 22:23:31 UTC
Features only compatible with hadoop 2.x
Hi,
While developing the out-of-core state backend that will store state
directly to hdfs (either TFiles or BloomMapFiles), I realised that some
file formats and features I use are hadoop 2.x only.
What is the suggested way to handle features that use hadoop 2.x api? Can
these be excluded from the travis build for the hadoop 1 profile somehow?
Thanks,
Gyula
Re: Features only compatible with hadoop 2.x
Posted by Stephan Ewen <se...@apache.org>.
At some point, the state backends should probably have their own parent-
and subprojects, similar to the batch-connectors and streaming-connectors.
Then you have the flexibility to include and exclude them as needed.
Stephan
On Sat, Jan 23, 2016 at 6:42 PM, Gyula Fóra <gy...@gmail.com> wrote:
> Alright, thanks.
>
> My dilemma is that currently all the out-of-core state stuff I put under
> streaming-contrib. So I either make the whole contrib package hadoop 2 only
> which is probably kind of awkward, or I move the main interfaces from
> contrib to somewhere else then I can put the HDFS backend stuff into a
> module that is hadoop 2 only.
>
> Gyula
>
> Robert Metzger <rm...@apache.org> ezt írta (időpont: 2016. jan. 22., P,
> 22:31):
>
> > Hi Gyula,
> >
> > will the out of core backend be in a separate maven module? If so, you
> can
> > include the module only in the "hadoop-2" profile.
> >
> > As you can see in the main pom.xml, "flink-yarn" and "flink-fs-tests"
> are
> > also "hadoop2" only modules:
> >
> > <profile>
> > <id>hadoop-2</id>
> > <activation>
> > <property>
> > <!-- Please do not remove the 'hadoop2' comment. See
> > ./tools/generate_specific_pom.sh -->
> > <!--hadoop2--><name>!hadoop.profile</name>
> > </property>
> > </activation>
> > <properties>
> > <hadoop.version>${hadoop-two.version}</hadoop.version>
> > <shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name
> >
> > <shading-artifact-module.name>flink-shaded-hadoop2</
> > shading-artifact-module.name>
> > </properties>
> > <modules>
> > <module>flink-yarn</module>
> > <!-- Include the flink-fs-tests project only for HD2.
> > The HDFS minicluster interfaces changed between the two
> versions.
> > -->
> > <module>flink-fs-tests</module>
> > </modules>
> > </profile>
> >
> >
> > If the backend is not in a separate maven module, you can use reflection.
> > Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush"
> > only if the method is available ;)
> >
> >
> >
> > On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <gy...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > While developing the out-of-core state backend that will store state
> > > directly to hdfs (either TFiles or BloomMapFiles), I realised that some
> > > file formats and features I use are hadoop 2.x only.
> > >
> > > What is the suggested way to handle features that use hadoop 2.x api?
> Can
> > > these be excluded from the travis build for the hadoop 1 profile
> somehow?
> > >
> > > Thanks,
> > > Gyula
> > >
> >
>
Re: Features only compatible with hadoop 2.x
Posted by Gyula Fóra <gy...@gmail.com>.
Alright, thanks.
My dilemma is that currently all the out-of-core state stuff I put under
streaming-contrib. So I either make the whole contrib package hadoop 2 only
which is probably kind of awkward, or I move the main interfaces from
contrib to somewhere else then I can put the HDFS backend stuff into a
module that is hadoop 2 only.
Gyula
Robert Metzger <rm...@apache.org> ezt írta (időpont: 2016. jan. 22., P,
22:31):
> Hi Gyula,
>
> will the out of core backend be in a separate maven module? If so, you can
> include the module only in the "hadoop-2" profile.
>
> As you can see in the main pom.xml, "flink-yarn" and "flink-fs-tests" are
> also "hadoop2" only modules:
>
> <profile>
> <id>hadoop-2</id>
> <activation>
> <property>
> <!-- Please do not remove the 'hadoop2' comment. See
> ./tools/generate_specific_pom.sh -->
> <!--hadoop2--><name>!hadoop.profile</name>
> </property>
> </activation>
> <properties>
> <hadoop.version>${hadoop-two.version}</hadoop.version>
> <shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name>
> <shading-artifact-module.name>flink-shaded-hadoop2</
> shading-artifact-module.name>
> </properties>
> <modules>
> <module>flink-yarn</module>
> <!-- Include the flink-fs-tests project only for HD2.
> The HDFS minicluster interfaces changed between the two versions.
> -->
> <module>flink-fs-tests</module>
> </modules>
> </profile>
>
>
> If the backend is not in a separate maven module, you can use reflection.
> Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush"
> only if the method is available ;)
>
>
>
> On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <gy...@apache.org> wrote:
>
> > Hi,
> >
> > While developing the out-of-core state backend that will store state
> > directly to hdfs (either TFiles or BloomMapFiles), I realised that some
> > file formats and features I use are hadoop 2.x only.
> >
> > What is the suggested way to handle features that use hadoop 2.x api? Can
> > these be excluded from the travis build for the hadoop 1 profile somehow?
> >
> > Thanks,
> > Gyula
> >
>
Re: Features only compatible with hadoop 2.x
Posted by Robert Metzger <rm...@apache.org>.
Hi Gyula,
will the out of core backend be in a separate maven module? If so, you can
include the module only in the "hadoop-2" profile.
As you can see in the main pom.xml, "flink-yarn" and "flink-fs-tests" are
also "hadoop2" only modules:
<profile>
<id>hadoop-2</id>
<activation>
<property>
<!-- Please do not remove the 'hadoop2' comment. See
./tools/generate_specific_pom.sh -->
<!--hadoop2--><name>!hadoop.profile</name>
</property>
</activation>
<properties>
<hadoop.version>${hadoop-two.version}</hadoop.version>
<shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name>
<shading-artifact-module.name>flink-shaded-hadoop2</shading-artifact-module.name>
</properties>
<modules>
<module>flink-yarn</module>
<!-- Include the flink-fs-tests project only for HD2.
The HDFS minicluster interfaces changed between the two versions.
-->
<module>flink-fs-tests</module>
</modules>
</profile>
If the backend is not in a separate maven module, you can use reflection.
Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush"
only if the method is available ;)
On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <gy...@apache.org> wrote:
> Hi,
>
> While developing the out-of-core state backend that will store state
> directly to hdfs (either TFiles or BloomMapFiles), I realised that some
> file formats and features I use are hadoop 2.x only.
>
> What is the suggested way to handle features that use hadoop 2.x api? Can
> these be excluded from the travis build for the hadoop 1 profile somehow?
>
> Thanks,
> Gyula
>