You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/08/18 05:24:17 UTC
[GitHub] [incubator-uniffle] zuston opened a new issue, #160: Support lower Hadoop versions in client-mr
zuston opened a new issue, #160:
URL: https://github.com/apache/incubator-uniffle/issues/160
Currently, uniffle use the default Hadoop version of 2.8.5.
When using the `./build_distribution.sh --spark2-profile 'spark2' --spark3-mvn '-Dspark.version=2.4.3' --spark3-profile 'spark3' --spark3-mvn '-Dspark.version=3.1.1' -Dhadoop.version=2.6.0`, it will throw exceptions due to some methods and vars not supported in Hadoop 2.6.0.
Some non-compatible params and methods as follows
1. `CallContext`, introduced by >= 2.8.0.
2. `MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENT` introduced by 2.8.0. [ticket link](https://issues.apache.org/jira/browse/MAPREDUCE-6105)
3. `MRApps.getSystemPropertiesToLog` introduced by 2.8.0 [ticket link](https://issues.apache.org/jira/browse/MAPREDUCE-5232)
I think we could use the reflection to be compatible with lower hadoop version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #160: Support lower Hadoop versions in client-mr
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #160:
URL: https://github.com/apache/incubator-uniffle/issues/160#issuecomment-1219113427
What's your company's hadoop version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #160: Support lower Hadoop versions in client-mr
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #160:
URL: https://github.com/apache/incubator-uniffle/issues/160#issuecomment-1219052007
cc @jerqi @frankliee
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #160: Support lower Hadoop versions in client-mr
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #160:
URL: https://github.com/apache/incubator-uniffle/issues/160#issuecomment-1219069364
And I also found the client-mr is not compatible with Hadoop 3.2.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #160: Support lower Hadoop versions in client-mr
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #160:
URL: https://github.com/apache/incubator-uniffle/issues/160#issuecomment-1219323340
OK. I will go ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #160: Support lower Hadoop versions in client-mr
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #160:
URL: https://github.com/apache/incubator-uniffle/issues/160#issuecomment-1219139826
I think it's ok for me if we need it in our production environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #160: Support lower Hadoop versions in client-mr
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #160:
URL: https://github.com/apache/incubator-uniffle/issues/160#issuecomment-1219117303
1. 2.6.0-cdh5.11.0
2. Hadoop-3.2.2 packaged by bigtop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org