[jira] [Resolved] (PHOENIX-1041) Document how to build phoenix with hadoop/hbase from CDH

["James Taylor (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PHOENIX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

James Taylor resolved PHOENIX-1041.
-----------------------------------
    Resolution: Won't Fix

Given that Phoenix is in Cloudera Labs, this seems unnecessary.

> Document how to build phoenix with hadoop/hbase from CDH
> --------------------------------------------------------
>
>                 Key: PHOENIX-1041
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-1041
>             Project: Phoenix
>          Issue Type: Improvement
>            Reporter: alex kamil
>            Priority: Minor
>
> Pls add instructions on building phoenix with hadoop/hbase from CDH. CDH deployments are very common and it's a nightmare to debug , e.g. in case hadoop version in phoenix doesn't match the one deployed on the cluster (e.g. see PHOENIX-56 - Phoenix/JDBC becomes unresponsive and PHOENIX-417 - Hadoop 2 woes) 
> This example is for phoenix 3.1 and CDH4.6.0: hadoop-2.0.0-cdh4.6.0 and hbase-2.0.0-cdh4.6.0
> 1. phoenix/pom.xml changes:
> add cloudera repository
>     <repository>
>       <id>cloudera</id>
>       <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
>     </repository>
> 2. update hadoop.profile  version
> <!-- Hadoop Versions -->
>     <hadoop-one.version>1.0.4</hadoop-one.version>
>     <!--hadoop-two.version>2.0.4-alpha</hadoop-two.version-->
>     <hadoop-two.version>2.0.0-cdh4.6.0</hadoop-two.version>
>  3. update dependency versions
> <!-- Dependency versions -->
>     <!--hbase.version>0.94.15</hbase.version-->
>     <hbase.version>0.94.15-cdh4.6.0</hbase.version>
>     <!--hadoop.version>2.0.0</hadoop.version-->   
>     <hadoop.version>2.0.0-cdh4.6.0</hadoop.version>
> 4. build phoenix with hadoop2 profile
>  mvn clean install -DskipTests -Dhadoop.profile=2
> 5. stop hadoop and hbase cluster if running
> 6. copy phoenix/phoenix-core/target/phoenix-core-3.1.0-snapshot.jar to each hadoop/hbase node in the cluster (e.g. to /opt/myapps/ directory)
> 7. add phoenix-core jar to hbase classpath
> update /etc/hbase/conf/hbase-env.sh:     
>                                                                                        
> export HBASE_CLASSPATH_PREFIX=/opt/myapps/phoenix-core-3.1.0-snapshot.jar  
> (Note: if I'm using HBASE_CLASSPATH getting SLF4J multiple bindings error, this is avoided by loading phoenix jar before the rest of the hbase dependencies with HBASE_CLASSPATH_PREFIX)
> 8. (optional) add phoenix-core jar to hadoop classpath
> update /etc/hadoop/conf/hadoop-env.sh (create the file in hadoop conf directory if doesn't exist)
> export HADOOP_CLASSPATH=/opt/myapps/phoenix-core-3.1.0-snapshot.jar 
> 9. start hadoop/hbase cluster
> check classpath:
> hbase classpath|tr ":" "\n"|grep phoenix
> create a test table with hbase shell:
> hbase shell
> hbase>create 't1',{NAME => 'f1',VERSIONS => 5}
> check hbase/hadoop/zk logs for errors
> start sqlline:
> 10. enable DEBUG logging in sqlline log4j.properties
> (otherwise sqlline is just "hanging"in case  of hadoop version mismatch and hides the related error:  java.lang.NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Ljava/io/InputStream, java.io.IOException: Could not set up IO Streams, client.HConnectionManager$HConnectionImplementation: getMaster attempt 8 of 14 failed )
> >save a backup copy of sqlline log4j.properties
> >replace WARN and ERROR with DEBUG in phoenix/bin/log4j.properties
> 11. add commons-collections, hadoop-common, hadoop-auth  and hbase  jars  from CDH to sqlline classpath 
> (Otherwise getting java.lang.NoClassDefFoundError: org/apache/commons/collections/map/UnmodifiableMap exception)
> update phoenix/bin/sqlline.py:
> extrajars= "/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
> cdh4.6.0.jar:/usr/lib/hbase/hbase-0.94.15-cdh4.6.0.jar"                                                                                                                                    
> java_cmd = 'java -classpath ".' + os.pathsep +extrajars+ os.pathsep+ phoenix_utils.phoenix_client_jar + \  
> 12. add /etc/hbase/conf/hbase-site.xml from one of the hbase nodes to phoenix/bin/
> 13. test
> phoenix/bin/sqlline.py <my_zookeeper_host>
> 14. building JDBC java client 
> >same jars as in step 11 need to be added to the classpath of a phoenix JDBC client:
> /usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-
> cdh4.6.0.jar:/usr/lib/hbase/hbase-0.94.15-cdh4.6.0.jar
> >enable DEBUG logging:
> add log4j.properties file to /src/main/resources/
> add to main:
> import org.apache.log4j.Logger;
> import org.apache.log4j.Level;
> ..
> Logger.getRootLogger().setLevel(Level.DEBUG);
> >add hbase-site.xml from one of the hbase nodes to to /src/main/resources/
> >if building with gradle:
> add to build.gradle:
> repositories { maven { url "https://repository.cloudera.com/artifactory/cloudera-repos/" }
> }
> compile(group: 'org.apache.hadoop', name:'hadoop-common',version:'2.0.0-alpha')
> compile(group: 'org.apache.hadoop', name: 'hadoop-auth', version:'2.0.0-alpha')
> compile(group: 'org.apache.hbase', name: 'hbase', version:'0.94.15')



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)