You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "alex kamil (JIRA)" <ji...@apache.org> on 2014/06/12 01:19:01 UTC
[jira] [Updated] (PHOENIX-1041) Building phoenix with hadoop/hbase
from CDH
[ https://issues.apache.org/jira/browse/PHOENIX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
alex kamil updated PHOENIX-1041:
--------------------------------
Description:
Pls add instructions on building phoenix with hadoop/hbase from CDH. CDH deployments are very common and it's a nightmare to debug.
This example is for CDH4.6.0: hadoop-2.0.0-cdh4.6.0 and hbase-2.0.0-cdh4.6.0:
1. phoenix/pom.xml changes:
add cloudera repository
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
2. update hadoop.profile version
<!-- Hadoop Versions -->
<hadoop-one.version>1.0.4</hadoop-one.version>
<!--hadoop-two.version>2.0.4-alpha</hadoop-two.version-->
<hadoop-two.version>2.0.0-cdh4.6.0</hadoop-two.version>
3. update dependency versions
<!-- Dependency versions -->
<!--hbase.version>0.94.15</hbase.version-->
<hbase.version>0.94.15-cdh4.6.0</hbase.version>
<!--hadoop.version>2.0.0</hadoop.version-->
<hadoop.version>2.0.0-cdh4.6.0</hadoop.version>
4. build phoenix with hadoop2 profile
mvn clean install -DskipTests -Dhadoop.profile=2
5. stop hadoop and hbase cluster if running
6. copy phoenix/phoenix-core/target/phoenix-core-3.1.0-snapshot.jar to each hadoop/hbase node in the cluster (e.g. to /opt/myapps/ directory)
7. add phoenix-core jar to hbase classpath
update /etc/hbase/conf/hbase-env.sh:
export HBASE_CLASSPATH_PREFIX=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
(Note: if I'm using HBASE_CLASSPATH getting SLF4J multiple bindings error, this is avoided by loading phoenix jar before the rest of the hbase dependencies with HBASE_CLASSPATH_PREFIX)
8. add phoenix-core jar to hadoop classpath
update /etc/hadoop/conf/hadoop-env.sh (create the file in hadoop conf directory if doesn't exist)
export HADOOP_CLASSPATH=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
9. start hadoop/hbase cluster
start sqlline:
10. enable DEBUG logging in sqlline log4j.properties
(otherwise sqlline is just "hanging"in case of hadoop version mismatch and hides the related error: DEBUG: NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream)
replace WARN and ERROR with DEBUG in phoenix/bin/log4j.properties
11. add commons-collections, hadoop-common and hadoop-auth jars to sqlline classpath
(Otherwise getting java.lang.NoClassDefFoundError: org/apache/commons/collections/map/UnmodifiableMap exception)
update phoenix/bin/sqlline.py:
extrajars= "/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
cdh4.6.0.jar"
java_cmd = 'java -classpath ".' + os.pathsep +extrajars+ os.pathsep+ phoenix_utils.phoenix_client_jar + \
12. test
phoenix/bin/sqlline.py <my_zookeeper_host>
was:
Pls add instructions on building phoenix with hadoop/hbase from CDH. CDH deployments are very common and it's a nightmare to debug.
This example is for CDH4.6.0: hadoop-2.0.0-cdh4.6.0 and hbase-2.0.0-cdh4.6.0:
1. phoenix/pom.xml changes:
add cloudera repository
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
2. update hadoop.profile version
<!-- Hadoop Versions -->
<hadoop-one.version>1.0.4</hadoop-one.version>
<!--hadoop-two.version>2.0.4-alpha</hadoop-two.version-->
<hadoop-two.version>2.0.0-cdh4.6.0</hadoop-two.version>
3. update dependency versions
<!-- Dependency versions -->
<!--hbase.version>0.94.15</hbase.version-->
<hbase.version>0.94.15-cdh4.6.0</hbase.version>
<!--hadoop.version>2.0.0</hadoop.version-->
<hadoop.version>2.0.0-cdh4.6.0</hadoop.version>
4. build phoenix with hadoop2 profile
mvn clean install -DskipTests -Dhadoop.profile=2
5. stop hadoop and hbase cluster if running
6. copy phoenix/phoenix-core/target/phoenix-core-3.1.0-snapshot.jar to each hadoop/hbase node in the cluster (e.g. to /opt/myapps/ directory)
7. add phoenix-core jar to hbase classpath
update /etc/hbase/conf/hbase-env.sh:
export HBASE_CLASSPATH_PREFIX=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
(Note: if I'm using HBASE_CLASSPATH getting SLF4J multiple bindings error, this is avoided by loading phoenix jar before the rest of the hbase dependencies with HBASE_CLASSPATH_PREFIX)
8. add phoenix-core jar to hadoop classpath
update /etc/hadoop/conf/hadoop-env.sh (create the file in hadoop conf directory if doesn't exist)
export HADOOP_CLASSPATH=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
NOTE:
9. start hadoop/hbase cluster
start sqlline:
10. enable DEBUG logging in sqlline log4j.properties
(otherwise sqlline is just "hanging"in case of hadoop version mismatch and hides the related error: DEBUG: NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream)
replace WARN and ERROR with DEBUG in phoenix/bin/log4j.properties
11. add commons-collections, hadoop-common and hadoop-auth jars to sqlline classpath
(Otherwise getting java.lang.NoClassDefFoundError: org/apache/commons/collections/map/UnmodifiableMap exception)
update phoenix/bin/sqlline.py:
extrajars= "/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
cdh4.6.0.jar"
java_cmd = 'java -classpath ".' + os.pathsep +extrajars+ os.pathsep+ phoenix_utils.phoenix_client_jar + \
12. test
phoenix/bin/sqlline.py <my_zookeeper_host>
> Building phoenix with hadoop/hbase from CDH
> -------------------------------------------
>
> Key: PHOENIX-1041
> URL: https://issues.apache.org/jira/browse/PHOENIX-1041
> Project: Phoenix
> Issue Type: Improvement
> Reporter: alex kamil
> Priority: Minor
>
> Pls add instructions on building phoenix with hadoop/hbase from CDH. CDH deployments are very common and it's a nightmare to debug.
> This example is for CDH4.6.0: hadoop-2.0.0-cdh4.6.0 and hbase-2.0.0-cdh4.6.0:
> 1. phoenix/pom.xml changes:
> add cloudera repository
> <repository>
> <id>cloudera</id>
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
> </repository>
> 2. update hadoop.profile version
> <!-- Hadoop Versions -->
> <hadoop-one.version>1.0.4</hadoop-one.version>
> <!--hadoop-two.version>2.0.4-alpha</hadoop-two.version-->
> <hadoop-two.version>2.0.0-cdh4.6.0</hadoop-two.version>
> 3. update dependency versions
> <!-- Dependency versions -->
> <!--hbase.version>0.94.15</hbase.version-->
> <hbase.version>0.94.15-cdh4.6.0</hbase.version>
> <!--hadoop.version>2.0.0</hadoop.version-->
> <hadoop.version>2.0.0-cdh4.6.0</hadoop.version>
> 4. build phoenix with hadoop2 profile
> mvn clean install -DskipTests -Dhadoop.profile=2
> 5. stop hadoop and hbase cluster if running
> 6. copy phoenix/phoenix-core/target/phoenix-core-3.1.0-snapshot.jar to each hadoop/hbase node in the cluster (e.g. to /opt/myapps/ directory)
> 7. add phoenix-core jar to hbase classpath
> update /etc/hbase/conf/hbase-env.sh:
>
> export HBASE_CLASSPATH_PREFIX=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
> (Note: if I'm using HBASE_CLASSPATH getting SLF4J multiple bindings error, this is avoided by loading phoenix jar before the rest of the hbase dependencies with HBASE_CLASSPATH_PREFIX)
> 8. add phoenix-core jar to hadoop classpath
> update /etc/hadoop/conf/hadoop-env.sh (create the file in hadoop conf directory if doesn't exist)
> export HADOOP_CLASSPATH=/opt/myapps/phoenix-core-3.1.0-snapshot.jar
> 9. start hadoop/hbase cluster
> start sqlline:
> 10. enable DEBUG logging in sqlline log4j.properties
> (otherwise sqlline is just "hanging"in case of hadoop version mismatch and hides the related error: DEBUG: NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream)
> replace WARN and ERROR with DEBUG in phoenix/bin/log4j.properties
> 11. add commons-collections, hadoop-common and hadoop-auth jars to sqlline classpath
> (Otherwise getting java.lang.NoClassDefFoundError: org/apache/commons/collections/map/UnmodifiableMap exception)
> update phoenix/bin/sqlline.py:
> extrajars= "/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
> cdh4.6.0.jar"
> java_cmd = 'java -classpath ".' + os.pathsep +extrajars+ os.pathsep+ phoenix_utils.phoenix_client_jar + \
> 12. test
> phoenix/bin/sqlline.py <my_zookeeper_host>
--
This message was sent by Atlassian JIRA
(v6.2#6252)