You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Zhang Xiaoyu <zh...@gmail.com> on 2013/10/10 21:09:47 UTC
is that possible to use Hive 0.11 with libthrift 0.7
Hi, all,
I am writing a piece of code talking to Hive 0.11 Hive Server 2. The JDBC
code depends on libthrift 0.9. However one component which depends on
libthrift 0.7 and not binary compatible with libthrift 0.9.
When I downgrade to 0.7, I got below NoClassDefFoundError:
org/apache/thrift/scheme/StandardScheme.
Any idea how to make HS2 working with thrift 0.7? Thanks.
Johnny
Re: is that possible to use Hive 0.11 with libthrift 0.7
Posted by Zhang Xiaoyu <zh...@gmail.com>.
Thanks, Edward. it is a big upgrade for other component. I guess I have to
got with class loader for each class for now.
Johnny
On Thu, Oct 10, 2013 at 12:19 PM, Edward Capriolo <ed...@gmail.com>wrote:
> You are kinda screwed. Thrift is wire-compatible in many cases but not API
> compatible. You can not have two applications build off two versions of
> thrift in the same classpath without something like OSGI. To insulate the
> class loaders from each other.
>
> Your best bet is upgrading "other component" to thrift 9
>
>
> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <zh...@gmail.com>wrote:
>
>> Hi, all,
>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The JDBC
>> code depends on libthrift 0.9. However one component which depends on
>> libthrift 0.7 and not binary compatible with libthrift 0.9.
>>
>> When I downgrade to 0.7, I got below NoClassDefFoundError:
>> org/apache/thrift/scheme/StandardScheme.
>>
>> Any idea how to make HS2 working with thrift 0.7? Thanks.
>>
>> Johnny
>>
>
>
Fwd: is that possible to use Hive 0.11 with libthrift 0.7
Posted by Timothy Potter <th...@gmail.com>.
Hi Johnny,
Sorry for the delayed response (was offline for a few days). So the code
that actually uses Thrift in is the Hive core JAR, which is what will get
shaded. The easiest thing to do is add this target to the root build.xml
file:
<target name="jarjar">
<taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
classpath="./jarjar-1.4.jar"/>
<jarjar jarfile="hive-jdbc-shaded.jar">
<zipgroupfileset dir="${hive.root}/build/dist/lib" includes="*.jar"/>
<rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1"/>
</jarjar>
</target>
Notice this assumes the JAR file for the jarjar task is in the hive.root
too. Run ant jarjar after doing a full ant clean package, which creates the
${hive.root}/build/dist/lib/*.jar
The jarjar task will produce a shaded JAR named hive-jdbc-shaded.jar that
contains all the hive jars and their dependencies with Thrift shaded into
the org.shaded.thrift package. Your client application needs to include
hive-jdbc-shaded.jar in its classpath (probably at the front).
Cheers,
Tim
On Thu, Oct 10, 2013 at 8:19 PM, Zhang Xiaoyu <zh...@gmail.com>wrote:
> Hi, Timothy,
> Thanks a lot for your reply. Appreciate it!
> I am facing the exact situation as yours. in one project, Cassandra client
> api need thrift 0.7 and Hive JDBC needs thrift 0.9......
>
> I am looking at the shade maven plugin, sorry I still have some silly
> questions, thanks a lot for your patience...
>
> I think you suggest two steps
> (1) need to include all of the dependencies of the Hive JDBC driver
> classes in a single JAR
> (2) the apply the shading the unified JAR file
>
> for *(1)* *instead* of putting the dependencies under <dependencies> tag,
> I now do this in the <plugins> section
>
> <plugin>
>
> <groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-shade-plugin</artifactId>
>
> <executions>
>
> <execution>
>
> <phase>package</phase>
>
> <goals>
>
> <goal>shade</goal>
>
> </goals>
>
> <configuration>
>
> <artifactSet>
>
> <includes>
>
> <include>org.apache.hive:hive-exec</include>
>
> <include>org.apache.hive:hive-jdbc</include>
>
> <include>org.apache.hive:hive-metastore</include>
>
> <include>org.apache.hive:hive-service</include>
>
> <include>org.apache.thrift:libfb303</include>
>
> <include>commons-logging:commons-logging</include>
>
> <include>org.apache.hadoop:hadoop-core</include>
>
> </includes>
>
> </artifactSet>
>
> </configuration>
>
> </execution>
>
> </executions>
>
> </plugin>
>
>
> question here is how do I specify the version for each jars? for example I
> want hive-exec 0.9, where to put version number ?
>
> what will this shaded jar looks like , I search the ~/.m2/ folder, didn't
> find a jar with proper name.
>
> *for (2) *not sure how to do it, I mean how to specify only the Hive JDBC
> class refer to this shaded jar ? but Cassandra client API class doesn't use
> it?
>
> Sorry again for so many silly questions, thanks a lot for your time!
>
> Johnny
>
>
> On Thu, Oct 10, 2013 at 5:12 PM, Timothy Potter <th...@gmail.com>wrote:
>
>> Sorry, sent too fast ... either way you do it, you'll need to include all
>> of the dependencies of the Hive JDBC driver classes in a single JAR and the
>> apply the shading the unified JAR file. This ensures that all classes that
>> depend on Thrift 0.9 get updated to use the shaded package. For example,
>> I'm working with the HCatalog Pig Adapter so I built a single JAR made up
>> of:
>>
>> antlr-runtime-3.4.jar commons-io-2.4.jar hive-beeline-0.11.0.jar
>> hive-serde-0.11.0.jar jdo2-api-2.3-ec.jar
>> protobuf-java-2.4.1.jar
>> avro-1.7.1.jar commons-lang-2.4.jar hive-cli-0.11.0.jar
>> hive-service-0.11.0.jar jetty-6.1.26.jar
>> servlet-api-2.5-20081211.jar
>> avro-mapred-1.7.1.jar commons-logging-1.0.4.jar
>> hive-common-0.11.0.jar hive-shims-0.11.0.jar
>> jetty-util-6.1.26.jar slf4j-api-1.6.1.jar
>> commons-cli-1.2.jar commons-logging-api-1.0.4.jar hive-contrib-0.11.0.jar
>> jackson-core-asl-1.8.8.jar jline-0.9.94.jar
>> slf4j-log4j12-1.6.1.jar
>> commons-codec-1.4.jar commons-pool-1.5.4.jar hive-exec-0.11.0.jar
>> jackson-jaxrs-1.8.8.jar json-20090211.jar snappy-0.2.jar
>> commons-collections-3.2.1.jar derby-10.4.2.0.jar hive-hbase-handler-0.11.0.jar
>> jackson-mapper-asl-1.8.8.jar libfb303-0.9.0.jar ST4-4.0.4.jar
>> commons-compress-1.4.1.jar guava-11.0.2.jar hive-hwi-0.11.0.jar
>> jackson-xc-1.8.8.jar libthrift-0.9.0.jar tempus-fugit-1.1.jar
>> commons-configuration-1.6.jar hcatalog-core-0.11.0.jar
>> hive-jdbc-0.11.0.jar JavaEWAH-0.3.2.jar log4j-1.2.16.jar
>> xz-1.0.jar
>> commons-dbcp-1.4.jar hcatalog-pig-adapter-0.11.0.jar hive-metastore-0.11.0.jar
>> javolution-5.5.1.jar metrics-core-2.1.2.jar zookeeper-3.4.3.jar
>>
>> Any class in any of these JAR files that depend on Thrift 0.9 classes
>> will get updated at the bytecode level to refer to my shaded package -
>> org.shaded.thrift
>>
>> Make sense?
>>
>>
>> On Thu, Oct 10, 2013 at 6:04 PM, Timothy Potter <th...@gmail.com>wrote:
>>
>>> I'm not sure about keep, but if you're going to use maven, go with the
>>> shade plugin vs. jarjar ... it does the same thing but has better maven
>>> integration - http://maven.apache.org/plugins/maven-shade-plugin/
>>>
>>>
>>>
>>>
>>> On Thu, Oct 10, 2013 at 5:31 PM, Zhang Xiaoyu <zh...@gmail.com>wrote:
>>>
>>>> Hi, Timothy,
>>>> Thanks for your reply. Seems jarjar is a solution for me. I have a
>>>> basic question to follow:
>>>>
>>>> I don't quite understand what is tag <keep> for ?
>>>>
>>>> My understand is
>>>> 1.
>>>> use <include> to grab the maven dependencies I want to re-package,
>>>> 2.
>>>> then use <pattern> and <result> in <rule> to define what class to
>>>> re-package and what's the new name for re-packaged binary.
>>>> 3.
>>>> And use <keep> to indicate which class in the project will only use
>>>> this re-packaged dependencies, in my case, it is JDBC class. Am I right ?
>>>>
>>>> below pom doesn't work for me, and JDBC class and other class still
>>>> share the same thrift 0.9. Thrift 0.7 is omitted.
>>>>
>>>> Thanks a lot for help,
>>>> Johnny
>>>>
>>>> <dependency>
>>>>
>>>> <groupId>org.apache.thrift</groupId>
>>>>
>>>> <artifactId>libthrift</artifactId>
>>>>
>>>> <version>0.9.0</version>
>>>>
>>>> </dependency>
>>>>
>>>> ......
>>>>
>>>> ......
>>>>
>>>> <plugin>
>>>>
>>>> <groupId>org.sonatype.plugins</groupId>
>>>>
>>>> <artifactId>jarjar-maven-plugin</artifactId>
>>>>
>>>> <executions>
>>>>
>>>> <execution>
>>>>
>>>> <phase>package</phase>
>>>>
>>>> <goals>
>>>>
>>>> <goal>jarjar</goal>
>>>>
>>>> </goals>
>>>>
>>>> <configuration>
>>>>
>>>> <includes>
>>>>
>>>> <include>*org.apache.thrift:libthrift*</include>
>>>>
>>>> </includes>
>>>>
>>>> <rules>
>>>>
>>>> <rule>
>>>>
>>>> <pattern>org.apache.thrift.**</pattern>
>>>>
>>>> <result>com.xxx.JDBC.class.pattern
>>>> .internal.thrift.@1</result>
>>>>
>>>> </rule>
>>>>
>>>> <keep>
>>>>
>>>> <pattern>com.xxx.JDBC.class.pattern.**</pattern>
>>>>
>>>> </keep>
>>>>
>>>> </rules>
>>>>
>>>> </configuration>
>>>>
>>>> </execution>
>>>>
>>>> </executions>
>>>>
>>>> </plugin>
>>>>
>>>>
>>>> On Thu, Oct 10, 2013 at 12:28 PM, Timothy Potter <th...@gmail.com>wrote:
>>>>
>>>>> Hi Zhang,
>>>>>
>>>>> I have the same issue in that I use some Cassandra client API's that
>>>>> depend on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted
>>>>> for using the jarjar utility to "shade" the thrift 0.9 classes. Here's what
>>>>> I added to the build.xml file for the hcatalog-pig-adapter project:
>>>>>
>>>>> <target name="jarjar">
>>>>> <echo message="basedir=${basedir}"/>
>>>>> <copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar"
>>>>> todir="${basedir}/deps"/>
>>>>> <copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar"
>>>>> todir="${basedir}/deps"/>
>>>>> <copy todir="${basedir}/deps">
>>>>> <fileset dir="${basedir}/../../build/dist/lib"
>>>>> includes="hive-*.jar"/>
>>>>> </copy>
>>>>> <taskdef name="jarjar"
>>>>> classname="com.tonicsystems.jarjar.JarJarTask"
>>>>> classpath="lib/jarjar-1.4.jar"/>
>>>>> <jarjar jarfile="hcat-pig-shaded.jar">
>>>>> <zipgroupfileset dir="deps" includes="*.jar"/>
>>>>> <rule pattern="org.apache.thrift.**"
>>>>> result="org.shaded.thrift.@1"/>
>>>>> </jarjar>
>>>>> </target>
>>>>>
>>>>> I basically cooked up a deps directory that contains all the JARs that
>>>>> the hcatalog-pig-adapter depends on and then used jarjar to create a single
>>>>> jar with the thrift classes renamed at the bytecode level to
>>>>> org.shaded.thrift
>>>>>
>>>>> You can do something similar for the JDBC code.
>>>>>
>>>>> Cheers,
>>>>> Tim
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo <
>>>>> edlinuxguru@gmail.com> wrote:
>>>>>
>>>>>> You are kinda screwed. Thrift is wire-compatible in many cases but
>>>>>> not API compatible. You can not have two applications build off two
>>>>>> versions of thrift in the same classpath without something like OSGI. To
>>>>>> insulate the class loaders from each other.
>>>>>>
>>>>>> Your best bet is upgrading "other component" to thrift 9
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <
>>>>>> zhangxiaoyu912@gmail.com> wrote:
>>>>>>
>>>>>>> Hi, all,
>>>>>>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The
>>>>>>> JDBC code depends on libthrift 0.9. However one component which depends on
>>>>>>> libthrift 0.7 and not binary compatible with libthrift 0.9.
>>>>>>>
>>>>>>> When I downgrade to 0.7, I got below NoClassDefFoundError:
>>>>>>> org/apache/thrift/scheme/StandardScheme.
>>>>>>>
>>>>>>> Any idea how to make HS2 working with thrift 0.7? Thanks.
>>>>>>>
>>>>>>> Johnny
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: is that possible to use Hive 0.11 with libthrift 0.7
Posted by Timothy Potter <th...@gmail.com>.
Sorry, sent too fast ... either way you do it, you'll need to include all
of the dependencies of the Hive JDBC driver classes in a single JAR and the
apply the shading the unified JAR file. This ensures that all classes that
depend on Thrift 0.9 get updated to use the shaded package. For example,
I'm working with the HCatalog Pig Adapter so I built a single JAR made up
of:
antlr-runtime-3.4.jar commons-io-2.4.jar hive-beeline-0.11.0.jar
hive-serde-0.11.0.jar jdo2-api-2.3-ec.jar protobuf-java-2.4.1.jar
avro-1.7.1.jar commons-lang-2.4.jar hive-cli-0.11.0.jar
hive-service-0.11.0.jar jetty-6.1.26.jar
servlet-api-2.5-20081211.jar
avro-mapred-1.7.1.jar commons-logging-1.0.4.jar hive-common-0.11.0.jar
hive-shims-0.11.0.jar jetty-util-6.1.26.jar slf4j-api-1.6.1.jar
commons-cli-1.2.jar commons-logging-api-1.0.4.jar hive-contrib-0.11.0.jar
jackson-core-asl-1.8.8.jar jline-0.9.94.jar
slf4j-log4j12-1.6.1.jar
commons-codec-1.4.jar commons-pool-1.5.4.jar hive-exec-0.11.0.jar
jackson-jaxrs-1.8.8.jar json-20090211.jar snappy-0.2.jar
commons-collections-3.2.1.jar derby-10.4.2.0.jar hive-hbase-handler-0.11.0.jar
jackson-mapper-asl-1.8.8.jar libfb303-0.9.0.jar ST4-4.0.4.jar
commons-compress-1.4.1.jar guava-11.0.2.jar hive-hwi-0.11.0.jar
jackson-xc-1.8.8.jar libthrift-0.9.0.jar tempus-fugit-1.1.jar
commons-configuration-1.6.jar hcatalog-core-0.11.0.jar hive-jdbc-0.11.0.jar
JavaEWAH-0.3.2.jar log4j-1.2.16.jar xz-1.0.jar
commons-dbcp-1.4.jar hcatalog-pig-adapter-0.11.0.jar
hive-metastore-0.11.0.jar
javolution-5.5.1.jar metrics-core-2.1.2.jar zookeeper-3.4.3.jar
Any class in any of these JAR files that depend on Thrift 0.9 classes will
get updated at the bytecode level to refer to my shaded package -
org.shaded.thrift
Make sense?
On Thu, Oct 10, 2013 at 6:04 PM, Timothy Potter <th...@gmail.com>wrote:
> I'm not sure about keep, but if you're going to use maven, go with the
> shade plugin vs. jarjar ... it does the same thing but has better maven
> integration - http://maven.apache.org/plugins/maven-shade-plugin/
>
>
>
>
> On Thu, Oct 10, 2013 at 5:31 PM, Zhang Xiaoyu <zh...@gmail.com>wrote:
>
>> Hi, Timothy,
>> Thanks for your reply. Seems jarjar is a solution for me. I have a basic
>> question to follow:
>>
>> I don't quite understand what is tag <keep> for ?
>>
>> My understand is
>> 1.
>> use <include> to grab the maven dependencies I want to re-package,
>> 2.
>> then use <pattern> and <result> in <rule> to define what class to
>> re-package and what's the new name for re-packaged binary.
>> 3.
>> And use <keep> to indicate which class in the project will only use this
>> re-packaged dependencies, in my case, it is JDBC class. Am I right ?
>>
>> below pom doesn't work for me, and JDBC class and other class still share
>> the same thrift 0.9. Thrift 0.7 is omitted.
>>
>> Thanks a lot for help,
>> Johnny
>>
>> <dependency>
>>
>> <groupId>org.apache.thrift</groupId>
>>
>> <artifactId>libthrift</artifactId>
>>
>> <version>0.9.0</version>
>>
>> </dependency>
>>
>> ......
>>
>> ......
>>
>> <plugin>
>>
>> <groupId>org.sonatype.plugins</groupId>
>>
>> <artifactId>jarjar-maven-plugin</artifactId>
>>
>> <executions>
>>
>> <execution>
>>
>> <phase>package</phase>
>>
>> <goals>
>>
>> <goal>jarjar</goal>
>>
>> </goals>
>>
>> <configuration>
>>
>> <includes>
>>
>> <include>*org.apache.thrift:libthrift*</include>
>>
>> </includes>
>>
>> <rules>
>>
>> <rule>
>>
>> <pattern>org.apache.thrift.**</pattern>
>>
>> <result>com.xxx.JDBC.class.pattern.internal.thrift.@1
>> </result>
>>
>> </rule>
>>
>> <keep>
>>
>> <pattern>com.xxx.JDBC.class.pattern.**</pattern>
>>
>> </keep>
>>
>> </rules>
>>
>> </configuration>
>>
>> </execution>
>>
>> </executions>
>>
>> </plugin>
>>
>>
>> On Thu, Oct 10, 2013 at 12:28 PM, Timothy Potter <th...@gmail.com>wrote:
>>
>>> Hi Zhang,
>>>
>>> I have the same issue in that I use some Cassandra client API's that
>>> depend on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted
>>> for using the jarjar utility to "shade" the thrift 0.9 classes. Here's what
>>> I added to the build.xml file for the hcatalog-pig-adapter project:
>>>
>>> <target name="jarjar">
>>> <echo message="basedir=${basedir}"/>
>>> <copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar"
>>> todir="${basedir}/deps"/>
>>> <copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar"
>>> todir="${basedir}/deps"/>
>>> <copy todir="${basedir}/deps">
>>> <fileset dir="${basedir}/../../build/dist/lib"
>>> includes="hive-*.jar"/>
>>> </copy>
>>> <taskdef name="jarjar"
>>> classname="com.tonicsystems.jarjar.JarJarTask"
>>> classpath="lib/jarjar-1.4.jar"/>
>>> <jarjar jarfile="hcat-pig-shaded.jar">
>>> <zipgroupfileset dir="deps" includes="*.jar"/>
>>> <rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1
>>> "/>
>>> </jarjar>
>>> </target>
>>>
>>> I basically cooked up a deps directory that contains all the JARs that
>>> the hcatalog-pig-adapter depends on and then used jarjar to create a single
>>> jar with the thrift classes renamed at the bytecode level to
>>> org.shaded.thrift
>>>
>>> You can do something similar for the JDBC code.
>>>
>>> Cheers,
>>> Tim
>>>
>>>
>>>
>>> On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>>
>>>> You are kinda screwed. Thrift is wire-compatible in many cases but not
>>>> API compatible. You can not have two applications build off two versions of
>>>> thrift in the same classpath without something like OSGI. To insulate the
>>>> class loaders from each other.
>>>>
>>>> Your best bet is upgrading "other component" to thrift 9
>>>>
>>>>
>>>> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <zhangxiaoyu912@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi, all,
>>>>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The
>>>>> JDBC code depends on libthrift 0.9. However one component which depends on
>>>>> libthrift 0.7 and not binary compatible with libthrift 0.9.
>>>>>
>>>>> When I downgrade to 0.7, I got below NoClassDefFoundError:
>>>>> org/apache/thrift/scheme/StandardScheme.
>>>>>
>>>>> Any idea how to make HS2 working with thrift 0.7? Thanks.
>>>>>
>>>>> Johnny
>>>>>
>>>>
>>>>
>>>
>>
>
Re: is that possible to use Hive 0.11 with libthrift 0.7
Posted by Timothy Potter <th...@gmail.com>.
I'm not sure about keep, but if you're going to use maven, go with the
shade plugin vs. jarjar ... it does the same thing but has better maven
integration - http://maven.apache.org/plugins/maven-shade-plugin/
On Thu, Oct 10, 2013 at 5:31 PM, Zhang Xiaoyu <zh...@gmail.com>wrote:
> Hi, Timothy,
> Thanks for your reply. Seems jarjar is a solution for me. I have a basic
> question to follow:
>
> I don't quite understand what is tag <keep> for ?
>
> My understand is
> 1.
> use <include> to grab the maven dependencies I want to re-package,
> 2.
> then use <pattern> and <result> in <rule> to define what class to
> re-package and what's the new name for re-packaged binary.
> 3.
> And use <keep> to indicate which class in the project will only use this
> re-packaged dependencies, in my case, it is JDBC class. Am I right ?
>
> below pom doesn't work for me, and JDBC class and other class still share
> the same thrift 0.9. Thrift 0.7 is omitted.
>
> Thanks a lot for help,
> Johnny
>
> <dependency>
>
> <groupId>org.apache.thrift</groupId>
>
> <artifactId>libthrift</artifactId>
>
> <version>0.9.0</version>
>
> </dependency>
>
> ......
>
> ......
>
> <plugin>
>
> <groupId>org.sonatype.plugins</groupId>
>
> <artifactId>jarjar-maven-plugin</artifactId>
>
> <executions>
>
> <execution>
>
> <phase>package</phase>
>
> <goals>
>
> <goal>jarjar</goal>
>
> </goals>
>
> <configuration>
>
> <includes>
>
> <include>*org.apache.thrift:libthrift*</include>
>
> </includes>
>
> <rules>
>
> <rule>
>
> <pattern>org.apache.thrift.**</pattern>
>
> <result>com.xxx.JDBC.class.pattern.internal.thrift.@1</
> result>
>
> </rule>
>
> <keep>
>
> <pattern>com.xxx.JDBC.class.pattern.**</pattern>
>
> </keep>
>
> </rules>
>
> </configuration>
>
> </execution>
>
> </executions>
>
> </plugin>
>
>
> On Thu, Oct 10, 2013 at 12:28 PM, Timothy Potter <th...@gmail.com>wrote:
>
>> Hi Zhang,
>>
>> I have the same issue in that I use some Cassandra client API's that
>> depend on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted
>> for using the jarjar utility to "shade" the thrift 0.9 classes. Here's what
>> I added to the build.xml file for the hcatalog-pig-adapter project:
>>
>> <target name="jarjar">
>> <echo message="basedir=${basedir}"/>
>> <copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar"
>> todir="${basedir}/deps"/>
>> <copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar"
>> todir="${basedir}/deps"/>
>> <copy todir="${basedir}/deps">
>> <fileset dir="${basedir}/../../build/dist/lib"
>> includes="hive-*.jar"/>
>> </copy>
>> <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
>> classpath="lib/jarjar-1.4.jar"/>
>> <jarjar jarfile="hcat-pig-shaded.jar">
>> <zipgroupfileset dir="deps" includes="*.jar"/>
>> <rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1
>> "/>
>> </jarjar>
>> </target>
>>
>> I basically cooked up a deps directory that contains all the JARs that
>> the hcatalog-pig-adapter depends on and then used jarjar to create a single
>> jar with the thrift classes renamed at the bytecode level to
>> org.shaded.thrift
>>
>> You can do something similar for the JDBC code.
>>
>> Cheers,
>> Tim
>>
>>
>>
>> On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>
>>> You are kinda screwed. Thrift is wire-compatible in many cases but not
>>> API compatible. You can not have two applications build off two versions of
>>> thrift in the same classpath without something like OSGI. To insulate the
>>> class loaders from each other.
>>>
>>> Your best bet is upgrading "other component" to thrift 9
>>>
>>>
>>> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <zh...@gmail.com>wrote:
>>>
>>>> Hi, all,
>>>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The
>>>> JDBC code depends on libthrift 0.9. However one component which depends on
>>>> libthrift 0.7 and not binary compatible with libthrift 0.9.
>>>>
>>>> When I downgrade to 0.7, I got below NoClassDefFoundError:
>>>> org/apache/thrift/scheme/StandardScheme.
>>>>
>>>> Any idea how to make HS2 working with thrift 0.7? Thanks.
>>>>
>>>> Johnny
>>>>
>>>
>>>
>>
>
Re: is that possible to use Hive 0.11 with libthrift 0.7
Posted by Zhang Xiaoyu <zh...@gmail.com>.
Hi, Timothy,
Thanks for your reply. Seems jarjar is a solution for me. I have a basic
question to follow:
I don't quite understand what is tag <keep> for ?
My understand is
1.
use <include> to grab the maven dependencies I want to re-package,
2.
then use <pattern> and <result> in <rule> to define what class to
re-package and what's the new name for re-packaged binary.
3.
And use <keep> to indicate which class in the project will only use this
re-packaged dependencies, in my case, it is JDBC class. Am I right ?
below pom doesn't work for me, and JDBC class and other class still share
the same thrift 0.9. Thrift 0.7 is omitted.
Thanks a lot for help,
Johnny
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
<version>0.9.0</version>
</dependency>
......
......
<plugin>
<groupId>org.sonatype.plugins</groupId>
<artifactId>jarjar-maven-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>jarjar</goal>
</goals>
<configuration>
<includes>
<include>*org.apache.thrift:libthrift*</include>
</includes>
<rules>
<rule>
<pattern>org.apache.thrift.**</pattern>
<result>com.xxx.JDBC.class.pattern.internal.thrift.@1</
result>
</rule>
<keep>
<pattern>com.xxx.JDBC.class.pattern.**</pattern>
</keep>
</rules>
</configuration>
</execution>
</executions>
</plugin>
On Thu, Oct 10, 2013 at 12:28 PM, Timothy Potter <th...@gmail.com>wrote:
> Hi Zhang,
>
> I have the same issue in that I use some Cassandra client API's that
> depend on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted
> for using the jarjar utility to "shade" the thrift 0.9 classes. Here's what
> I added to the build.xml file for the hcatalog-pig-adapter project:
>
> <target name="jarjar">
> <echo message="basedir=${basedir}"/>
> <copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar"
> todir="${basedir}/deps"/>
> <copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar"
> todir="${basedir}/deps"/>
> <copy todir="${basedir}/deps">
> <fileset dir="${basedir}/../../build/dist/lib"
> includes="hive-*.jar"/>
> </copy>
> <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
> classpath="lib/jarjar-1.4.jar"/>
> <jarjar jarfile="hcat-pig-shaded.jar">
> <zipgroupfileset dir="deps" includes="*.jar"/>
> <rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1
> "/>
> </jarjar>
> </target>
>
> I basically cooked up a deps directory that contains all the JARs that the
> hcatalog-pig-adapter depends on and then used jarjar to create a single jar
> with the thrift classes renamed at the bytecode level to org.shaded.thrift
>
> You can do something similar for the JDBC code.
>
> Cheers,
> Tim
>
>
>
> On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> You are kinda screwed. Thrift is wire-compatible in many cases but not
>> API compatible. You can not have two applications build off two versions of
>> thrift in the same classpath without something like OSGI. To insulate the
>> class loaders from each other.
>>
>> Your best bet is upgrading "other component" to thrift 9
>>
>>
>> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <zh...@gmail.com>wrote:
>>
>>> Hi, all,
>>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The
>>> JDBC code depends on libthrift 0.9. However one component which depends on
>>> libthrift 0.7 and not binary compatible with libthrift 0.9.
>>>
>>> When I downgrade to 0.7, I got below NoClassDefFoundError:
>>> org/apache/thrift/scheme/StandardScheme.
>>>
>>> Any idea how to make HS2 working with thrift 0.7? Thanks.
>>>
>>> Johnny
>>>
>>
>>
>
Re: is that possible to use Hive 0.11 with libthrift 0.7
Posted by Timothy Potter <th...@gmail.com>.
Hi Zhang,
I have the same issue in that I use some Cassandra client API's that depend
on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted for
using the jarjar utility to "shade" the thrift 0.9 classes. Here's what I
added to the build.xml file for the hcatalog-pig-adapter project:
<target name="jarjar">
<echo message="basedir=${basedir}"/>
<copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar"
todir="${basedir}/deps"/>
<copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar"
todir="${basedir}/deps"/>
<copy todir="${basedir}/deps">
<fileset dir="${basedir}/../../build/dist/lib" includes="hive-*.jar"/>
</copy>
<taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
classpath="lib/jarjar-1.4.jar"/>
<jarjar jarfile="hcat-pig-shaded.jar">
<zipgroupfileset dir="deps" includes="*.jar"/>
<rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1"/>
</jarjar>
</target>
I basically cooked up a deps directory that contains all the JARs that the
hcatalog-pig-adapter depends on and then used jarjar to create a single jar
with the thrift classes renamed at the bytecode level to org.shaded.thrift
You can do something similar for the JDBC code.
Cheers,
Tim
On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo <ed...@gmail.com>wrote:
> You are kinda screwed. Thrift is wire-compatible in many cases but not API
> compatible. You can not have two applications build off two versions of
> thrift in the same classpath without something like OSGI. To insulate the
> class loaders from each other.
>
> Your best bet is upgrading "other component" to thrift 9
>
>
> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <zh...@gmail.com>wrote:
>
>> Hi, all,
>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The JDBC
>> code depends on libthrift 0.9. However one component which depends on
>> libthrift 0.7 and not binary compatible with libthrift 0.9.
>>
>> When I downgrade to 0.7, I got below NoClassDefFoundError:
>> org/apache/thrift/scheme/StandardScheme.
>>
>> Any idea how to make HS2 working with thrift 0.7? Thanks.
>>
>> Johnny
>>
>
>
Re: is that possible to use Hive 0.11 with libthrift 0.7
Posted by Edward Capriolo <ed...@gmail.com>.
You are kinda screwed. Thrift is wire-compatible in many cases but not API
compatible. You can not have two applications build off two versions of
thrift in the same classpath without something like OSGI. To insulate the
class loaders from each other.
Your best bet is upgrading "other component" to thrift 9
On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <zh...@gmail.com>wrote:
> Hi, all,
> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The JDBC
> code depends on libthrift 0.9. However one component which depends on
> libthrift 0.7 and not binary compatible with libthrift 0.9.
>
> When I downgrade to 0.7, I got below NoClassDefFoundError:
> org/apache/thrift/scheme/StandardScheme.
>
> Any idea how to make HS2 working with thrift 0.7? Thanks.
>
> Johnny
>