You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "Roman Shaposhnik (Created) (JIRA)" <ji...@apache.org> on 2012/02/29 23:21:59 UTC
[jira] [Created] (BIGTOP-423) hadoop package needs to be split into
hadoop-client and hadoop-server packages
hadoop package needs to be split into hadoop-client and hadoop-server packages
------------------------------------------------------------------------------
Key: BIGTOP-423
URL: https://issues.apache.org/jira/browse/BIGTOP-423
Project: Bigtop
Issue Type: Bug
Components: General
Affects Versions: 0.4.0
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Fix For: 0.4.0
Currently hadoop package co-bundles together dependencies for the daemons of hadoop (HDFS, YARN) and client side of the same projects. It would be much nicer to split this functionality into 2 separate packages so that downstream components (Pig,Hive,Oozie) don't have to depend on more bits than they have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-423) hadoop package needs to be split
into hadoop-client and hadoop-server packages
Posted by "Bruno Mahé (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228066#comment-13228066 ]
Bruno Mahé commented on BIGTOP-423:
-----------------------------------
Some notes:
* Could you attach hadoop-client.list so I can see what it looks like?
* Are you bumping hadoop version on purpose as part of this ticket?
* Wouldn't a "find" be more appropriate rather than this for/continue loop ?
* Requires in spec files can be on multiple lines. It would make it easier to read it if you could split the require
> hadoop package needs to be split into hadoop-client and hadoop-server packages
> ------------------------------------------------------------------------------
>
> Key: BIGTOP-423
> URL: https://issues.apache.org/jira/browse/BIGTOP-423
> Project: Bigtop
> Issue Type: Bug
> Components: General
> Affects Versions: 0.4.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: BIGTOP-423.patch.txt
>
>
> Currently hadoop package co-bundles together dependencies for the daemons of hadoop (HDFS, YARN) and client side of the same projects. It would be much nicer to split this functionality into 2 separate packages so that downstream components (Pig,Hive,Oozie) don't have to depend on more bits than they have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (BIGTOP-423) hadoop package needs to
be split into hadoop-client and hadoop-server packages
Posted by "Bruno Mahé (Issue Comment Edited JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228066#comment-13228066 ]
Bruno Mahé edited comment on BIGTOP-423 at 3/12/12 11:48 PM:
-------------------------------------------------------------
Some notes:
* Could you attach hadoop-client.list so I can see what it looks like?
* Are you bumping hadoop version on purpose as part of this ticket?
* Wouldn't a "find" be more appropriate rather than this for/continue loop ?
* In a spec file, "Requires" statements can be split on multiple lines. It would make it easier to read it if you could split the require
was (Author: bmahe):
Some notes:
* Could you attach hadoop-client.list so I can see what it looks like?
* Are you bumping hadoop version on purpose as part of this ticket?
* Wouldn't a "find" be more appropriate rather than this for/continue loop ?
* Requires in spec files can be on multiple lines. It would make it easier to read it if you could split the require
> hadoop package needs to be split into hadoop-client and hadoop-server packages
> ------------------------------------------------------------------------------
>
> Key: BIGTOP-423
> URL: https://issues.apache.org/jira/browse/BIGTOP-423
> Project: Bigtop
> Issue Type: Bug
> Components: General
> Affects Versions: 0.4.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: BIGTOP-423.patch.txt
>
>
> Currently hadoop package co-bundles together dependencies for the daemons of hadoop (HDFS, YARN) and client side of the same projects. It would be much nicer to split this functionality into 2 separate packages so that downstream components (Pig,Hive,Oozie) don't have to depend on more bits than they have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BIGTOP-423) hadoop package needs to be split into
hadoop-client and hadoop-server packages
Posted by "Roman Shaposhnik (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated BIGTOP-423:
------------------------------------
Priority: Blocker (was: Major)
> hadoop package needs to be split into hadoop-client and hadoop-server packages
> ------------------------------------------------------------------------------
>
> Key: BIGTOP-423
> URL: https://issues.apache.org/jira/browse/BIGTOP-423
> Project: Bigtop
> Issue Type: Bug
> Components: General
> Affects Versions: 0.4.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.4.0
>
>
> Currently hadoop package co-bundles together dependencies for the daemons of hadoop (HDFS, YARN) and client side of the same projects. It would be much nicer to split this functionality into 2 separate packages so that downstream components (Pig,Hive,Oozie) don't have to depend on more bits than they have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-423) hadoop package needs to be split
into hadoop-client and hadoop-server packages
Posted by "Roman Shaposhnik (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225793#comment-13225793 ]
Roman Shaposhnik commented on BIGTOP-423:
-----------------------------------------
Here's the current proposal (which is a bit less ambitious than the original intent since hadoop is still not fully ready to be cleanly split into server/client parts at the level of jar files):
# we are going to have an extra package called hadoop-client
# hadoop-client will depend on hadoop, hadoop-hdfs, hadoop-yarn and hadoop-mapreduce since they all have client and server bits co-mingled in their respective jar files
# hadoop-client package will install a bunch of the symbolic links under /usr/lib/hadoop/lib/client pointing to all the jar files in the packages it depends upon
That way we will have a single location to be added to the client's class path (/usr/lib/hadoop/lib/client) and we can slowly work on spliting the actual jar files between the client and server packages.
Thoughts?
> hadoop package needs to be split into hadoop-client and hadoop-server packages
> ------------------------------------------------------------------------------
>
> Key: BIGTOP-423
> URL: https://issues.apache.org/jira/browse/BIGTOP-423
> Project: Bigtop
> Issue Type: Bug
> Components: General
> Affects Versions: 0.4.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.4.0
>
>
> Currently hadoop package co-bundles together dependencies for the daemons of hadoop (HDFS, YARN) and client side of the same projects. It would be much nicer to split this functionality into 2 separate packages so that downstream components (Pig,Hive,Oozie) don't have to depend on more bits than they have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BIGTOP-423) hadoop package needs to be split into
hadoop-client and hadoop-server packages
Posted by "Roman Shaposhnik (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated BIGTOP-423:
------------------------------------
Attachment: BIGTOP-423.patch.txt
Attaching a patch that also requires moving us to the tip of the branch-0.23.
Please let me know what do you think.
Once we agree that this type of hadoop-client package looks good I can transition all the pig/hive/sqoop/etc dependencies onto it.
> hadoop package needs to be split into hadoop-client and hadoop-server packages
> ------------------------------------------------------------------------------
>
> Key: BIGTOP-423
> URL: https://issues.apache.org/jira/browse/BIGTOP-423
> Project: Bigtop
> Issue Type: Bug
> Components: General
> Affects Versions: 0.4.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: BIGTOP-423.patch.txt
>
>
> Currently hadoop package co-bundles together dependencies for the daemons of hadoop (HDFS, YARN) and client side of the same projects. It would be much nicer to split this functionality into 2 separate packages so that downstream components (Pig,Hive,Oozie) don't have to depend on more bits than they have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-423) hadoop package needs to be split
into hadoop-client and hadoop-server packages
Posted by "Roman Shaposhnik (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228107#comment-13228107 ]
Roman Shaposhnik commented on BIGTOP-423:
-----------------------------------------
@Bruno,
# I've attached the current version of the hadoop-client.list. The thing about it, of course, is that it has a flexibility of changing as Hadoop developers see fit -- hence we have to ask Hadoop build for it instead of maintaining our own copy.
# Correct, we need need the bump specifically because of this ticket. The hadoop-client is there and the fix for MAPREDUCE-3996 (although not the one that is proposed on that JIRA)
# Hm. Not sure about the find -- if you don't mind please paste the code you're thinking about and we can decide
# Will absolutely split Requires -- didn't know about that
Ok, at this point, barring your find suggestion I take it that the general idea is acceptable?
> hadoop package needs to be split into hadoop-client and hadoop-server packages
> ------------------------------------------------------------------------------
>
> Key: BIGTOP-423
> URL: https://issues.apache.org/jira/browse/BIGTOP-423
> Project: Bigtop
> Issue Type: Bug
> Components: General
> Affects Versions: 0.4.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: BIGTOP-423.patch.txt, hadoop-client.list
>
>
> Currently hadoop package co-bundles together dependencies for the daemons of hadoop (HDFS, YARN) and client side of the same projects. It would be much nicer to split this functionality into 2 separate packages so that downstream components (Pig,Hive,Oozie) don't have to depend on more bits than they have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BIGTOP-423) hadoop package needs to be split into
hadoop-client and hadoop-server packages
Posted by "Roman Shaposhnik (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated BIGTOP-423:
------------------------------------
Attachment: hadoop-client.list
> hadoop package needs to be split into hadoop-client and hadoop-server packages
> ------------------------------------------------------------------------------
>
> Key: BIGTOP-423
> URL: https://issues.apache.org/jira/browse/BIGTOP-423
> Project: Bigtop
> Issue Type: Bug
> Components: General
> Affects Versions: 0.4.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: BIGTOP-423.patch.txt, hadoop-client.list
>
>
> Currently hadoop package co-bundles together dependencies for the daemons of hadoop (HDFS, YARN) and client side of the same projects. It would be much nicer to split this functionality into 2 separate packages so that downstream components (Pig,Hive,Oozie) don't have to depend on more bits than they have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BIGTOP-423) hadoop package needs to be split
into hadoop-client and hadoop-server packages
Posted by "Bruno Mahé (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BIGTOP-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228840#comment-13228840 ]
Bruno Mahé commented on BIGTOP-423:
-----------------------------------
Thanks!
So given that :
* we are both aware the zookeeper jar wouldn't be picked up as is because of the versionless zk jar in hadoop. But you said you would fix that in a coming ticket
* I am too busy/lazy to come up with the right snippet for 3.
* You will fix the requires in an updated patch
+1
> hadoop package needs to be split into hadoop-client and hadoop-server packages
> ------------------------------------------------------------------------------
>
> Key: BIGTOP-423
> URL: https://issues.apache.org/jira/browse/BIGTOP-423
> Project: Bigtop
> Issue Type: Bug
> Components: General
> Affects Versions: 0.4.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: BIGTOP-423.patch.txt, hadoop-client.list
>
>
> Currently hadoop package co-bundles together dependencies for the daemons of hadoop (HDFS, YARN) and client side of the same projects. It would be much nicer to split this functionality into 2 separate packages so that downstream components (Pig,Hive,Oozie) don't have to depend on more bits than they have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira