Re: ycsb and hbase 0.95.1 hadoop2

[Paul Baclace <pa...@gmail.com>]

I eventually determined that bin/ycsb (a python script) was causing most 
of the linkage problems, so I recommend using your own bash script to 
start ycsb for HBase. The problem is that the ordering of paths in the 
classpath is completely uncontrolled: bin/ycsb does not put the 
hbase-binding/conf first, so it can pick up a default hbase-site.xml 
found in a jar, and it will recursively include any jars in any 
subdirectories under hbase-binding/ (a surprise when I mv'ed some files 
to a subdir like dont_use_these and lsof showed them to be in use).

I ended up using hbase 0.95.1 hadoop1 on Apache Hadoop 1.0.4 for my 
benchmarking.


On 20130729 13:34 , Nicolas Liochon wrote:
> 0.95.2 is not yet finished. As Ted suggested, you can build HBase locally
> from the sources (the 0.95 branch is usually good as the patches are tested
> before being committed).
>
> When 0.95.2 will be out I will redo the integration with ycsb. Hopefully it
> will be simple to make it work with all hadoop versions on the standards
> maven repo...
>
>
> On Mon, Jul 29, 2013 at 10:14 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> You can perform the following command in 0.95 workspace:
>>
>> mvn clean install -DskipTests
>>
>> Cheers
>>
>> On Mon, Jul 29, 2013 at 1:11 PM, Paul Baclace <paul.baclace@gmail.com
>>> wrote:
>>> On 20130729 12:18 , Nicolas Liochon wrote:
>>>
>>>> No, I've got nothing in my setting.xml
>>>> I build HBase locally (mvn install -Dhadoop.profile=2.0 -DskipTests), so
>>>> yes, it goes into my .m2
>>>>
>>>>
>>>> I've just rechecked with the last 0.97 as of now, it does compile. It
>>>> worked as well 2 weeks ago with the 0.95 branch. I'm not sure about the
>>>> 95.1. I remember some nasty maven issues.
>>>>
>>> ((I rather be attacked by angry ants than nasty mavens.))
>>>
>>>
>>>> If you're using the maven repo for the 0.95.1, IIRC there is no version
>>>> available that works well with hadoop 2 with a maven-built client (it's
>> ok
>>>> if you build hbase locally with the right hadoop version).
>>>>
>>> Is there an experimental or nightly 0.95.2 maven repo I can connect to? I
>>> can probably use 0.95.2 if all the tests passed; it looks like
>>> https://builds.apache.org/job/**HBase-0.95/373<
>> https://builds.apache.org/job/HBase-0.95/373>passed.
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jul 29, 2013 at 8:50 PM, Paul Baclace <paul.baclace@gmail.com
>>>>> wrote:
>>>>   Thanks for your help, Nicolas. I adjusted the params and I get a
>>>>> ycsb/db/HBaseClient.java compile error (below) without any missing
>>>>> package
>>>>> errors, so I'm wondering how it can work for you. I am building by
>> adding
>>>>> artifacts from 0.95.1 release to my .m2; and you are building a
>> snapshot
>>>>> and "installing" that to .m2 (or other repo), I assume.
>>>>>
>>>>> It appears to be missing hbase-common which I added to .m2 and the
>>>>> compiler debug shows a classpath that only contains
>>>>> hbase-server-0.95.1-hadoop2.****jar (plus ycsb and general jars). Does
>>>>> your
>>>>> mvn -X compile cmd line for ycsb/db/HBaseClient.java contain more hbase
>>>>> or
>>>>> hadoop jars? Do you have a settings.xml that makes mvn pull in more
>>>>> dependencies?
>>>>>
>>>>>
>>>>> [ERROR] YCSB/hbase/src/main/java/com/****yahoo/ycsb/db/HBaseClient.****
>>>>> java:[34,30]
>>>>> package org.apache.hadoop.conf does not exist
>>>>> [ERROR]
>> /YCSB/hbase/src/main/java/com/****yahoo/ycsb/db/HBaseClient.****
>>>>> java:[35,31]
>>>>> cannot find symbol
>>>>> symbol  : class KeyValue
>>>>>
>>>>>
>>>>>
>>>>> On 20130729 9:29 , Nicolas Liochon wrote:
>>>>>
>>>>>   You can use the branch master here:
>> https://github.com/nkeywal/****YCSB<https://github.com/nkeywal/**YCSB>
>>>>>> <https://github.com/**nkeywal/YCSB <https://github.com/nkeywal/YCSB>>
>>>>>> Built it with:
>>>>>> mvn clean package -DskipTests -Dhbase-96 -Dhbase.version=0.97.0-**
>>>>>> SNAPSHOT
>>>>>> -Dhadoop.profile=2.0 -Dhadoop-two.version=2.0.5
>>>>>> (obviously, change the hbase.version & hadoop-two.version as you
>> need).
>>>>>> I'm awaiting the finalized hadoop2 flags in hbase pom to update this
>> and
>>>>>> push it to the main ycsb repo.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Nicolas
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 29, 2013 at 6:04 PM, Nick Dimiduk <nd...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>    Hi Paul,
>>>>>>
>>>>>>> I think Nicolas Liochon maintains a personal set of branches for
>> this.
>>>>>>> Maybe he can point you in the right direction.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Nick
>>>>>>>
>>>>>>> On Mon, Jul 29, 2013 at 12:48 AM, Paul Baclace <
>> paul.baclace@gmail.com
>>>>>>>   wrote:
>>>>>>>> [This might look like a user list question, or a question for
>> another
>>>>>>>> project, but people on the hbase dev list use ycsb to test hbase,
>> and
>>>>>>>> the
>>>>>>>> side effect of my progress on this benefits hbase more than any
>> other
>>>>>>>> project... please indulge me.]
>>>>>>>>
>>>>>>>> I have been unable to build ycsb for hbase 0.95.1  hadoop2, even
>>>>>>>> though
>>>>>>>> I
>>>>>>>> added the artifacts to my local repo like so:
>>>>>>>>
>>>>>>>> mvn install:install-file -Dfile=../hbase-0.95.1-**
>>>>>>>> hadoop2/lib/hadoop-mapreduce-******client-core-2.0.2-alpha.jar
>>>>>>>> -DgroupId=org.apache.hadoop -DartifactId=hadoop-mapreduce-****
>>>>>>>> **client-core
>>>>>>>> -Dversion=2.0.2-alpha -Dpackaging=jar
>>>>>>>>
>>>>>>>> mvn install:install-file -Dfile=../hbase-0.95.1-**
>>>>>>>> hadoop2/lib/hbase-client-0.95.******1-hadoop2.jar
>>>>>>>> -DgroupId=org.apache.hbase
>>>>>>>> -DartifactId=hbase-client -Dversion=0.95.1-hadoop2 -Dpackaging=jar
>>>>>>>>
>>>>>>>> I looked at the pom.xml in the jar files to determine the
>> parameters,
>>>>>>>> modified ycsb pom.xml and hbase/pom.xml; ycsb  "mvn clean package"
>> has
>>>>>>>> errors, not finding basic stuff:
>>>>>>>>
>>>>>>>> [ERROR] YCSB/hbase/src/main/**
>>>>>>>>
>>>>>>>> java/com/yahoo/ycsb/db/******HBaseClient.java:[34,29] package
>>>>>>>> org.apache.hadoop.conf does not exist
>>>>>>>> [ERROR] YCSB/hbase/src/main/**
>>>>>>>>
>>>>>>>> java/com/yahoo/ycsb/db/******HBaseClient.java:[35,30] cannot find
>>>>>>>> symbol
>>>>>>>> symbol  : class KeyValue
>>>>>>>> location: package org.apache.hadoop.hbase
>>>>>>>> [ERROR] YCSB/hbase/src/main/**
>>>>>>>>
>>>>>>>> java/com/yahoo/ycsb/db/******HBaseClient.java:[46,35] cannot find
>>>>>>>> symbol
>>>>>>>> symbol  : class Bytes
>>>>>>>> location: package org.apache.hadoop.hbase.util
>>>>>>>> [ERROR] YCSB/hbase/src/main/**
>>>>>>>>
>>>>>>>> java/com/yahoo/ycsb/db/******HBaseClient.java:[47,30] cannot find
>>>>>>>> symbol
>>>>>>>> symbol  : class HBaseConfiguration
>>>>>>>> location: package org.apache.hadoop.hbase
>>>>>>>> [... more not shown for brevity ]
>>>>>>>>
>>>>>>>> I suspect ycsb pom.xml [https://github.com/******
>>>>>>>> brianfrankcooper/YCSB <https://github.com/****brianfrankcooper/YCSB
>>> <
>>>>>>>> https://**github.com/**brianfrankcooper/**YCSB<
>> https://github.com/**brianfrankcooper/YCSB>
>>>>>>>> <
>>>>>>>>
>>>>>>>>   https://github.com/****brianfrankcooper/YCSB<
>> https://github.com/**brianfrankcooper/YCSB>
>>>>>>> <https://**github.com/brianfrankcooper/**YCSB<
>> https://github.com/brianfrankcooper/YCSB>
>>>>>>>> ]
>>>>>>>> is not ready for packaging changes made in hbase 0.95 OR could it be
>>>>>>>> that
>>>>>>>> adding jars to a local maven repo, because they are not yet on a
>>>>>>>> centralized server, simply does not suffice because not all
>>>>>>>> dependencies
>>>>>>>> are added?   HBaseConfiguration is in
>> hbase-common-0.95.1-hadoop2.***
>>>>>>>> ***jar
>>>>>>>> ,
>>>>>>>> but how to add all the turtles all the way down?
>>>>>>>>
>>>>>>>> What's the secret to using ycsb for the latest HBase?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>