You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Steve Loughran <st...@cloudera.com.INVALID> on 2021/01/12 13:58:06 UTC
[DISCUSS] HADOOP-17464. Create hadoop-compression module
Following on from discussions in a previous PR, Liang-Chi Hsieh has created
a PR to add a new hadoop-compression module, with the goal being "new
compression codecs and their dependencies can go in here rather than
hadoop-common"
https://github.com/apache/hadoop/pull/2611
I think this is the right thing to do as it keeps the stuff out of
hadoop-common dependencies, and isolates the changes.
But as you can see from the test results on that PR, there's stuff in
hadoop-hdfs and hadoop-mapreduce which expects those codes to always been
on the classpath.
What to do?
1. We add the new pom as a dependency of hadoop hdfs server and its tests
(but not hdfs client), and for MR tests
2. we leave the old codecs in hadoop-common, and its only the recent stuff
which we add to the new module.
Suggestions?
Re: [DISCUSS] HADOOP-17464. Create hadoop-compression module
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
> But as you can see from the test results on that PR, there's stuff in
> hadoop-hdfs and hadoop-mapreduce which expects those codes to always been
> on the classpath.
It sounds like incompatible change of hadoop-common rather than
test scope issue of HDFS and MapReduce.
How about making hadoop-compression a compile scope dependency
of hadoop-common first then removing it on the next major release (4.0.0)?
Thanks,
Masatake Iwasaki
On 2021/01/12 22:58, Steve Loughran wrote:
> Following on from discussions in a previous PR, Liang-Chi Hsieh has created
> a PR to add a new hadoop-compression module, with the goal being "new
> compression codecs and their dependencies can go in here rather than
> hadoop-common"
>
> https://github.com/apache/hadoop/pull/2611
>
> I think this is the right thing to do as it keeps the stuff out of
> hadoop-common dependencies, and isolates the changes.
>
> But as you can see from the test results on that PR, there's stuff in
> hadoop-hdfs and hadoop-mapreduce which expects those codes to always been
> on the classpath.
>
> What to do?
>
> 1. We add the new pom as a dependency of hadoop hdfs server and its tests
> (but not hdfs client), and for MR tests
> 2. we leave the old codecs in hadoop-common, and its only the recent stuff
> which we add to the new module.
>
> Suggestions?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSS] HADOOP-17464. Create hadoop-compression module
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
> But as you can see from the test results on that PR, there's stuff in
> hadoop-hdfs and hadoop-mapreduce which expects those codes to always been
> on the classpath.
It sounds like incompatible change of hadoop-common rather than
test scope issue of HDFS and MapReduce.
How about making hadoop-compression a compile scope dependency
of hadoop-common first then removing it on the next major release (4.0.0)?
Thanks,
Masatake Iwasaki
On 2021/01/12 22:58, Steve Loughran wrote:
> Following on from discussions in a previous PR, Liang-Chi Hsieh has created
> a PR to add a new hadoop-compression module, with the goal being "new
> compression codecs and their dependencies can go in here rather than
> hadoop-common"
>
> https://github.com/apache/hadoop/pull/2611
>
> I think this is the right thing to do as it keeps the stuff out of
> hadoop-common dependencies, and isolates the changes.
>
> But as you can see from the test results on that PR, there's stuff in
> hadoop-hdfs and hadoop-mapreduce which expects those codes to always been
> on the classpath.
>
> What to do?
>
> 1. We add the new pom as a dependency of hadoop hdfs server and its tests
> (but not hdfs client), and for MR tests
> 2. we leave the old codecs in hadoop-common, and its only the recent stuff
> which we add to the new module.
>
> Suggestions?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org