You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Josh Wills (JIRA)" <ji...@apache.org> on 2012/07/06 18:08:33 UTC
[jira] [Created] (CRUNCH-4) Create profiles for Crunch builds based
on Hadoop 1.0 and Hadoop 2.0
Josh Wills created CRUNCH-4:
-------------------------------
Summary: Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
Key: CRUNCH-4
URL: https://issues.apache.org/jira/browse/CRUNCH-4
Project: Crunch
Issue Type: Task
Components: Core
Reporter: Josh Wills
Assignee: Josh Wills
We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
Reference:
http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423708#comment-13423708 ]
Josh Wills commented on CRUNCH-4:
---------------------------------
Hey Rahul-- the code changes look good, thank you. On the POM files, it looks like your editor changed all of the 2-space indents to tabs, which is making the diff difficult to parse-- could you change it back to the 2-space indents?
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422331#comment-13422331 ]
Josh Wills commented on CRUNCH-4:
---------------------------------
Very cool, I'll play with it this evening. I added Matthias to the watchers list, since I think he has some dependency cleanup work that will interact w/this change.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423993#comment-13423993 ]
Josh Wills commented on CRUNCH-4:
---------------------------------
hadoop 2.0.0-alpha works fine except for the fact that the only HBase version the tests pass with is 0.92.1-cdh4.0.0. It doesn't look like the HBase project has released a version yet that works with 2.0.0-alpha.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-4) Create profiles for Crunch builds based
on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-4:
------------------------------
Attachment: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch
fix updated
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-4) Create profiles for Crunch builds based
on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-4:
------------------------------
Attachment: 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch
I created profiles for CDH3/hbase .90 and hadoop2/hbase .92 . Also tests/srcs were modified for the same. By default CDH3 profile gets built. The other can be invoked by setting property crunch.platform=2
In hadoop 2 the Aggregate API returned output in reverse order. Also the Distributed Cache behavior was a bit different.
A hadoop2 run shows a bunch of deprecated warning eg:
42334 [Thread-288] WARN org.apache.hadoop.conf.Configuration - mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
42336 [Thread-288] WARN org.apache.hadoop.conf.Configuration - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
42337 [Thread-288] WARN org.apache.hadoop.conf.Configuration - mapred.output.key.comparator.class is deprecated. Instead, use mapreduce.job.output.key.comparator.class
42338 [Thread-288] WARN org.apache.hadoop.conf.Configuration - mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
42340 [Thread-288] WARN org.apache.hadoop.conf.Configuration - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
42342 [Thread-288] WARN org.apache.hadoop.conf.Configuration - mapred.job.name is deprecated. Instead, use mapreduce.job.name
42344 [Thread-288] WARN org.apache.hadoop.conf.Configuration - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
Should we try to fix them ?
I did not made a profile for hadoop1 as a version of the same could not be found in Cloudera repository.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Josh Wills
> Labels: maven, newbie
> Attachments: 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423871#comment-13423871 ]
Arun C Murthy commented on CRUNCH-4:
------------------------------------
Also, I don't see why Crunch should have a vendor distro-specific profiles i.e. the hadoop-CDH3 profile. Is the expectation that Crunch will have several of these and that you plan to maintain them?
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-4) Create profiles for Crunch builds based
on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-4:
------------------------------
Attachment: Removed-cloudera-repository.patch
Patch for removing Cloudera repository.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Fix For: 0.3.0
>
> Attachments: 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch, Removed-cloudera-repository.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424164#comment-13424164 ]
Josh Wills commented on CRUNCH-4:
---------------------------------
Waiting how? Meaning that they're not creating a hadoop-2 profile until hbase finishes their release?
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427899#comment-13427899 ]
Rahul Sharma commented on CRUNCH-4:
-----------------------------------
We will make a hadoop-2 profile after the release of hbase 0.94. But shall we use hadoop 1 version and discard the current 0.20.2-CDH one ?
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437323#comment-13437323 ]
Rahul Sharma commented on CRUNCH-4:
-----------------------------------
Matthias, I have attached the patch with the grant.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Fix For: 0.3.0
>
> Attachments: 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch, Removed-cloudera-repository.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424097#comment-13424097 ]
Josh Wills commented on CRUNCH-4:
---------------------------------
HBase developers are voting on a release that will work with hadoop 2.0.0: http://search-hadoop.com/m/OF1v71Jxybc2
I'm inclined to commit this and open a JIRA to track changing the hbase version for the hadoop-2 profile to be 0.94.1 when it goes out so that we can make progress on other things. We would not do a Crunch release that had a vendor-specific dependency for a core component. Any objections?
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422912#comment-13422912 ]
Rahul Sharma commented on CRUNCH-4:
-----------------------------------
The hadoop 1.0.3 profile prints a few stack traces like ( output from WordCountHBaseIT test) :
2940 [main] WARN org.apache.hadoop.metrics2.util.MBeans - Hadoop:service=DataNode,name=MetricsSystem,sub=Control
javax.management.InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
at com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
at com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
at com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
WARN org.apache.hadoop.metrics2.util.MBeans - Hadoop:service=DataNode,name=DataNodeInfo
javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=DataNodeInfo
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403)
at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506)
But these all are warning and are listed in HBASE-4709 issue, which is yet to be fixed. I think in future we need to spend some time in fixing them.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423974#comment-13423974 ]
Josh Wills commented on CRUNCH-4:
---------------------------------
I'm not sure that Crunch works w/hadoop 0.20.2-- there is some issue involving multiple output handling, IIRC. Will need to check it out.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423710#comment-13423710 ]
Rahul Sharma commented on CRUNCH-4:
-----------------------------------
Josh, I will fix it.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428171#comment-13428171 ]
Josh Wills commented on CRUNCH-4:
---------------------------------
It looks like the HBase folks are going to make an 0.94.1 release next week (August 7th is when the vote closes) which looks like it will fix the incompatibility between HBase testing and hadoop 2.0.0-alpha. I would like to wait for that release and try it out as our hadoop-2 HBase dependency in this patch. If it works, we're good to go. If it doesn't, we have a number of fairly unpleasant options to consider, e.g., dropping hadoop 2.0.0 support for this release, or writing our own version of the HBaseTestingUtility.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-4) Create profiles for Crunch builds based
on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-4:
------------------------------
Attachment: CRUNCH-4-Profiles-for-hadoop-verions.patch
Looking at ppl having some issues with hadoop 1.0.3 I thought it would better to have a profile for the same. The profile can be enabled using variable crunch.platform=1. The profile uses hadoop 1.0.3 and hbase 0.90.6-CDH3.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411641#comment-13411641 ]
Tom White commented on CRUNCH-4:
--------------------------------
MRUnit does this too - see http://svn.apache.org/viewvc/mrunit/trunk/pom.xml?view=markup for the code.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Josh Wills
> Labels: maven, newbie
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-4) Create profiles for Crunch builds based
on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-4:
------------------------------
Attachment: CRUNCH-4-Hadoop-Profiles.patch
Patch corrected for indentation
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424648#comment-13424648 ]
Arun C Murthy commented on CRUNCH-4:
------------------------------------
Yep, they've postponed the bigtop-0.4.0 release till then.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424162#comment-13424162 ]
Arun C Murthy commented on CRUNCH-4:
------------------------------------
I believe bigtop is awaiting the hbase release, maybe we can do the same here for the hadoop-2 profile?
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-4) Create profiles for Crunch builds based
on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-4:
----------------------------
Attachment: CRUNCH-4-No-Vendor-Specific-Profiles.patch
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437327#comment-13437327 ]
Matthias Friedrich commented on CRUNCH-4:
-----------------------------------------
Works on a fresh repository cache, all of our dependencies are available from Maven central. Committed, thank you!
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Fix For: 0.3.0
>
> Attachments: 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch, Removed-cloudera-repository.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423867#comment-13423867 ]
Arun C Murthy commented on CRUNCH-4:
------------------------------------
bq. Hey Rahul-- the code changes look good, thank you.
Rahul/Josh - I'm confused.
Crunch is now an Apache incubator project.
Hadoop, HBase etc. are Apache projects.
Why are we having Crunch dependencies based on CDH rather than the Apache releases?
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423966#comment-13423966 ]
Rahul Sharma commented on CRUNCH-4:
-----------------------------------
Good point guys, crunch should not depend on vendor specific platforms.
Also Josh, you siad that we need to keep hadoop 1.0.3 and 2.0.0 profiles. But in order to work with 0.20 version of hadoop we dropped BSON support in CRUNCH-20/ CRUNCH-16 issue. Do we enable than one also ? I mean when we fix CRUNCH-16 issue should be enable that.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423947#comment-13423947 ]
Josh Wills commented on CRUNCH-4:
---------------------------------
Rahul-- Arun is right, we don't want to be in the business of maintaining vendor-specific versions of Crunch, so we just need the hadoop-1.0.3 and hadoop-2.0.0-alpha based profiles (along with base HBase versions that are stable-- I think 0.90.4 and 0.92.1 will work). Apache Bigtop will take care of the vendor distributions stuff. I'll update the patch, find good HBase versions, and upload a copy of it before I commit it.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills reassigned CRUNCH-4:
-------------------------------
Assignee: Rahul Sharma (was: Josh Wills)
Re-assigning to Rahul, who has been working on this one.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13432265#comment-13432265 ]
Josh Wills commented on CRUNCH-4:
---------------------------------
So I downloaded the 0.94.1 HBase release and verified that Crunch patches tests with it-- *IF* you download the HBase source files and compile it yourself using:
mvn clean install -Dhadoop.profile=2.0 -DskipTests
and then run crunch as:
mvn clean verify -Dcrunch.platform=2
The default maven jars go against the 1.0.x version of Hadoop and will fail if you run them with Hadoop 2.0.0-alpha. I'm not sure of the best way to message that to people short of putting a note on the website, at least until the HBase guys release a maven version that is compiled against 2.0.0-alpha. In any case, I'd like to integrate Rahul's changes so that we can commit this, break the HBase stuff into its own subcomponent, finish CRUNCH-16, and release.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Resolved] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by Matthias Friedrich <ma...@mafr.de>.
Hi Rahul,
I would have committed it, but I think your patch should be attached
to CRUNCH-4 with the "Grant license" option checked. Apache
regulations :)
Thanks,
Matthias
On Thursday, 2012-08-16, Rahul wrote:
> Josh, I wasn't there for quite some time so replying on this quite
> late. There is just one small thing. I think we can drop cloudera
> repository from the parent pom.
> Since the issue is resolved, so I did know if it is a good idea to
> reopen it or create a new issue for the small update. Attached is
> the patch for the same.
>
> J
> On 13-08-2012 07:06, Josh Wills (JIRA) wrote:
> > [ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
> >
> >Josh Wills resolved CRUNCH-4.
> >-----------------------------
> >
> > Resolution: Fixed
> > Fix Version/s: 0.3.0
> >
> >Just committed this after including the workaround in HBASE-5711 for directory permissions on the HBase miniclusters. Rahul did the lion's share of the work here-- thanks Rahul, and thanks for your patience!
> >>Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> >>--------------------------------------------------------------------
> >>
> >> Key: CRUNCH-4
> >> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> >> Project: Crunch
> >> Issue Type: Task
> >> Components: Core
> >> Reporter: Josh Wills
> >> Assignee: Rahul Sharma
> >> Labels: maven, newbie
> >> Fix For: 0.3.0
> >>
> >> Attachments: 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
> >>
> >>
> >>We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> >>Reference:
> >>http://maven.apache.org/guides/introduction/introduction-to-profiles.html
> >--
> >This message is automatically generated by JIRA.
> >If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> >For more information on JIRA, see: http://www.atlassian.com/software/jira
> >
>
> From 6db97e5c66ef70d12c43800d7296920c375bf43c Mon Sep 17 00:00:00 2001
> From: Rahul Sharma <rs...@xebia.com>
> Date: Thu, 16 Aug 2012 09:57:22 +0530
> Subject: [PATCH] Removed cloudera repository
>
> ---
> pom.xml | 10 ----------
> 1 files changed, 0 insertions(+), 10 deletions(-)
>
> diff --git a/pom.xml b/pom.xml
> index 6fcb21f..424a643 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -269,16 +269,6 @@ under the License.
>
> </dependencyManagement>
>
> - <!-- TODO: After changing our Hadoop/HBase dependencies to Apache
> - releases we no longer need this section. -->
> - <repositories>
> - <repository>
> - <id>maven-hadoop</id>
> - <name>Hadoop Releases</name>
> - <url>https://repository.cloudera.com/content/repositories/releases/</url>
> - </repository>
> - </repositories>
> -
> <profiles>
> <profile>
> <id>hadoop-1</id>
> --
> 1.7.1
>
Re: [jira] [Resolved] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by Josh Wills <jw...@cloudera.com>.
Hey Rahul-- good catch. It should be a small no-op, so I'll submit it
tomorrow lest I break anything tonight (my time). ;-)
On Wed, Aug 15, 2012 at 9:38 PM, Rahul <rs...@xebia.com> wrote:
> Josh, I wasn't there for quite some time so replying on this quite late.
> There is just one small thing. I think we can drop cloudera repository from
> the parent pom.
> Since the issue is resolved, so I did know if it is a good idea to reopen it
> or create a new issue for the small update. Attached is the patch for the
> same.
>
> J
>
> On 13-08-2012 07:06, Josh Wills (JIRA) wrote:
>>
>> [
>> https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>> ]
>>
>> Josh Wills resolved CRUNCH-4.
>> -----------------------------
>>
>> Resolution: Fixed
>> Fix Version/s: 0.3.0
>>
>> Just committed this after including the workaround in HBASE-5711 for
>> directory permissions on the HBase miniclusters. Rahul did the lion's share
>> of the work here-- thanks Rahul, and thanks for your patience!
>>
>>>
>>> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
>>> --------------------------------------------------------------------
>>>
>>> Key: CRUNCH-4
>>> URL: https://issues.apache.org/jira/browse/CRUNCH-4
>>> Project: Crunch
>>> Issue Type: Task
>>> Components: Core
>>> Reporter: Josh Wills
>>> Assignee: Rahul Sharma
>>> Labels: maven, newbie
>>> Fix For: 0.3.0
>>>
>>> Attachments:
>>> 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch,
>>> 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch,
>>> CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch,
>>> CRUNCH-4-Profiles-for-hadoop-verions.patch
>>>
>>>
>>> We need to create two Maven build profiles for Crunch, one that depends
>>> on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop
>>> 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
>>> Reference:
>>> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
>>
>> --
>> This message is automatically generated by JIRA.
>> If you think it was sent incorrectly, please contact your JIRA
>> administrators:
>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>> For more information on JIRA, see: http://www.atlassian.com/software/jira
>>
>>
>
>
--
Director of Data Science
Cloudera
Twitter: @josh_wills
Re: [jira] [Resolved] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by Rahul <rs...@xebia.com>.
Josh, I wasn't there for quite some time so replying on this quite late.
There is just one small thing. I think we can drop cloudera repository
from the parent pom.
Since the issue is resolved, so I did know if it is a good idea to
reopen it or create a new issue for the small update. Attached is the
patch for the same.
J
On 13-08-2012 07:06, Josh Wills (JIRA) wrote:
> [ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Josh Wills resolved CRUNCH-4.
> -----------------------------
>
> Resolution: Fixed
> Fix Version/s: 0.3.0
>
> Just committed this after including the workaround in HBASE-5711 for directory permissions on the HBase miniclusters. Rahul did the lion's share of the work here-- thanks Rahul, and thanks for your patience!
>
>> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
>> --------------------------------------------------------------------
>>
>> Key: CRUNCH-4
>> URL: https://issues.apache.org/jira/browse/CRUNCH-4
>> Project: Crunch
>> Issue Type: Task
>> Components: Core
>> Reporter: Josh Wills
>> Assignee: Rahul Sharma
>> Labels: maven, newbie
>> Fix For: 0.3.0
>>
>> Attachments: 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>>
>>
>> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
>> Reference:
>> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
[jira] [Resolved] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills resolved CRUNCH-4.
-----------------------------
Resolution: Fixed
Fix Version/s: 0.3.0
Just committed this after including the workaround in HBASE-5711 for directory permissions on the HBase miniclusters. Rahul did the lion's share of the work here-- thanks Rahul, and thanks for your patience!
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Fix For: 0.3.0
>
> Attachments: 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch, 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, CRUNCH-4-Hadoop-Profiles.patch, CRUNCH-4-No-Vendor-Specific-Profiles.patch, CRUNCH-4-Profiles-for-hadoop-verions.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-4) Create profiles for Crunch builds
based on Hadoop 1.0 and Hadoop 2.0
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422095#comment-13422095 ]
Rahul Sharma commented on CRUNCH-4:
-----------------------------------
Fix has been updated for DistributedCache. In hadoop2 the cache is no longer build on /tmp folder. Its gets built on {$hadoop-job} folder. So using just file name rather than absolute path in MapsideJoins. Also fix has been made in sync with changes in master.
> Create profiles for Crunch builds based on Hadoop 1.0 and Hadoop 2.0
> --------------------------------------------------------------------
>
> Key: CRUNCH-4
> URL: https://issues.apache.org/jira/browse/CRUNCH-4
> Project: Crunch
> Issue Type: Task
> Components: Core
> Reporter: Josh Wills
> Assignee: Rahul Sharma
> Labels: maven, newbie
> Attachments: 0001-CRUNCH-4-Profiles-for-hadoop-0.20-and-2.0-versions.patch, 0001-Created-profiles-for-CDH3-hbase-.90.6-and-hadoop2-hb.patch
>
>
> We need to create two Maven build profiles for Crunch, one that depends on Hadoop 1.0 (probably 1.0.3 for Hadoop and 0.90.5 for HBase) and Hadoop 2.0 (2.0.0-alpha for Hadoop and a stable 0.92.x for HBase.)
> Reference:
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira