You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2023/12/18 09:02:40 UTC
[PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
zuston opened a new pull request, #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379
### What changes were proposed in this pull request?
provide distribution jars with hadoop
### Why are the changes needed?
Fix: #1369
1. For `MEMORY_LOCALFILE` mode, there is no need to set HADOOP_HOME, which is hard for some users to set up this environment.
2. Using the embedded hadoop jars, it could has the higher version of hadoop to use the latest features.
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
Internal tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862390727
> How about `-Phadoop-dependencies-included`
More reasonable. Ok for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862011315
Could you help review this? @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862029070
For point 1, I have some concern. If a user wants to use MEMORY_LOCALFILE, he should compile with the the profile with -PwithHadoop. I feel it's not a good name. I have no better ideas about the name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1859861846
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1379?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`ce102ce`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/ce102ce9fb31ce976a6268ca39467fac45c8507c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 53.23% compared to head [(`20cd868`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1379?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 54.13%.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #1379 +/- ##
============================================
+ Coverage 53.23% 54.13% +0.90%
Complexity 2717 2717
============================================
Files 419 399 -20
Lines 23977 21616 -2361
Branches 2045 2045
============================================
- Hits 12764 11702 -1062
+ Misses 10428 9200 -1228
+ Partials 785 714 -71
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1379?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862153091
> > > For point 1, I have some concern. If a user wants to use MEMORY_LOCALFILE, he should compile with the the profile with -Pwith-hadoop. It's a little weird. I feel it's not a good name. I have no better ideas about the name.
> >
> >
> > how about `hadoop-compile` ?
>
> Why do I use a profile whose name contains `hadoop` if I am a user of MEMORY_LOCALFILE?
Inaccurate. This is a convenient way for user to use MEMORY_LOCALFILE rather than only for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: Provide distribution with Hadoop dependencies [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862263774
> Spark also does the similar things. And it's not necessary to introduce dedicated profile for memory_localfile case.
Em.. I agree. But the name is a biggest problem. -Pwith-hadoop may be confusing. Because we have already had -Phadoop2.8 and -Phadoop3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862376768
How about `-Phadoop-dependencies-included`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "qqqttt123 (via GitHub)" <gi...@apache.org>.
qqqttt123 commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862201832
> > > > For point 1, I have some concern. If a user wants to use MEMORY_LOCALFILE, he should compile with the the profile with -Pwith-hadoop. It's a little weird. I feel it's not a good name. I have no better ideas about the name.
> > >
> > >
> > > how about `hadoop-compile` ?
> >
> >
> > Why do I use a profile whose name contains `hadoop` if I am a user of MEMORY_LOCALFILE?
>
> Inaccurate. This is a convenient way for user to use MEMORY_LOCALFILE rather than only for this
But it will cause the user who use MEMORY_LOCALFILE confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862071944
> > For point 1, I have some concern. If a user wants to use MEMORY_LOCALFILE, he should compile with the the profile with -Pwith-hadoop. It's a little weird. I feel it's not a good name. I have no better ideas about the name.
>
> how about `hadoop-compile` ?
Why do I use a profile whose name contains `hadoop` if I am a user of MEMORY_LOCALFILE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862203253
> > > > For point 1, I have some concern. If a user wants to use MEMORY_LOCALFILE, he should compile with the the profile with -Pwith-hadoop. It's a little weird. I feel it's not a good name. I have no better ideas about the name.
> > >
> > >
> > > how about `hadoop-compile` ?
> >
> >
> > Why do I use a profile whose name contains `hadoop` if I am a user of MEMORY_LOCALFILE?
>
> Inaccurate. This is a convenient way for user to use MEMORY_LOCALFILE rather than only for this
It will cause the users who use MEMORY_LOCALFILE confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862230382
Spark also does the similar things. And it's not necessary to introduce dedicated profile for memory_localfile case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1369] feat: provide distribution jars with hadoop [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1379:
URL: https://github.com/apache/incubator-uniffle/pull/1379#issuecomment-1862069325
> For point 1, I have some concern. If a user wants to use MEMORY_LOCALFILE, he should compile with the the profile with -Pwith-hadoop. It's a little weird. I feel it's not a good name. I have no better ideas about the name.
how about `hadoop-compile` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org