cassandra 0.8.4 + pig (using cloudera rpms)

[William Oberman <ob...@civicscience.com>]

I've had some troubles, so I thought I'd pass on my various bug fixes:

-Cass 0.8.4 has troubles with pig/hadoop (you get NPE's when trying to connect to cassandra in the pig logs).  You need this patch:
http://svn.apache.org/viewvc?revision=1158940&view=revision
And maybe this:
http://svn.apache.org/viewvc?revision=1155157&view=revision

-I had installed from riptano rpms.  I downloaded the src, applied the patch, and did "ant jar".  I then replaced the rpm installed cassandra jar with this new one (ugly, but I wanted to continue to run from the package).

-I think I was able to just replace the apache-cassandra-0.8.4.jar on just my jobtracker + tasktracker nodes (I need to retest from scratch to be sure, I've done a _lot_ of configuring and reconfiguring)

-Then I started getting ClassNotFound exceptions during map/reduce tasks.  Still not sure why this fix works, but the problem seems to be cloudera pig version 0.20.2+923.97-1 has two jars that match pig*.jar (which is what cassandra contrib/pig/bin/pig_cassandra uses to setup the classpath).  I had to rename /usr/lib/pig/pig.jar for things to work (leaving pig-0.8.1-cdh3u1-core.jar as the only match).

My pig script is still running, but it's the first time it didn't immediately crash.

will

Re: cassandra 0.8.4 + pig (using cloudera rpms)

[William Oberman <ob...@civicscience.com>]

Yes, my cluster is working.

I didn't realize it at the time, but the StorageService link I listed is
already in 0.8.4, so yes the only file I had to patch was VersionedValue.
 Not sure what was going on with the pig jars, but after more configuration
changes than I can count, I'm pretty sure removing pig.jar in favor of the
cloudera pig jar was the magic bullet (for the ClassNotFound I was getting,
in this case TException).

One final note: in production I had to patch all of my cassandra servers
(OLTP and analytics)* with the VersionedValue file for it to work (though, I
did forget one setting, so now I'm still not 100% sure I had to patch all of
them, but it's working now).

will

OLTP = vanilla cassandra
analytics = cassandra + tasktracker.  I'm not on brisk yet, so I've been
rolling my own.


On Mon, Sep 5, 2011 at 1:41 AM, Jeremy Hanna <je...@gmail.com>wrote:

> Thanks William - so you were able to get everything running correctly,
> right?
>
> FWIW, we're in the process of upgrading to 0.8.4 and found that all we
> needed was that first link you mentioned - the VersionedValue modification.
>  It's running fine on our staging cluster and we're in the process of moving
> to production.  We're currently using pig from cdhu0.  All we did was
> replace the 0.8.4 jars after installing the debian packages for 0.8.4.
>
> Not sure if that helps anyone, but thought I would share what we've seen.
>
> Btw, this shouldn't be a problem once 0.8.5 comes out.
>
> On Sep 4, 2011, at 11:03 AM, William Oberman wrote:
>
> > I've had some troubles, so I thought I'd pass on my various bug fixes:
> >
> > -Cass 0.8.4 has troubles with pig/hadoop (you get NPE's when trying to
> connect to cassandra in the pig logs).  You need this patch:
> > http://svn.apache.org/viewvc?revision=1158940&view=revision
> > And maybe this:
> > http://svn.apache.org/viewvc?revision=1155157&view=revision
> >
> > -I had installed from riptano rpms.  I downloaded the src, applied the
> patch, and did "ant jar".  I then replaced the rpm installed cassandra jar
> with this new one (ugly, but I wanted to continue to run from the package).
> >
> > -I think I was able to just replace the apache-cassandra-0.8.4.jar on
> just my jobtracker + tasktracker nodes (I need to retest from scratch to be
> sure, I've done a _lot_ of configuring and reconfiguring)
> >
> > -Then I started getting ClassNotFound exceptions during map/reduce tasks.
>  Still not sure why this fix works, but the problem seems to be cloudera pig
> version 0.20.2+923.97-1 has two jars that match pig*.jar (which is what
> cassandra contrib/pig/bin/pig_cassandra uses to setup the classpath).  I had
> to rename /usr/lib/pig/pig.jar for things to work (leaving
> pig-0.8.1-cdh3u1-core.jar as the only match).
> >
> > My pig script is still running, but it's the first time it didn't
> immediately crash.
> >
> > will
> >
> >
>
>

Re: cassandra 0.8.4 + pig (using cloudera rpms)

[Jeremy Hanna <je...@gmail.com>]

Thanks William - so you were able to get everything running correctly, right?

FWIW, we're in the process of upgrading to 0.8.4 and found that all we needed was that first link you mentioned - the VersionedValue modification.  It's running fine on our staging cluster and we're in the process of moving to production.  We're currently using pig from cdhu0.  All we did was replace the 0.8.4 jars after installing the debian packages for 0.8.4.

Not sure if that helps anyone, but thought I would share what we've seen.

Btw, this shouldn't be a problem once 0.8.5 comes out.

On Sep 4, 2011, at 11:03 AM, William Oberman wrote:

> I've had some troubles, so I thought I'd pass on my various bug fixes:
> 
> -Cass 0.8.4 has troubles with pig/hadoop (you get NPE's when trying to connect to cassandra in the pig logs).  You need this patch:
> http://svn.apache.org/viewvc?revision=1158940&view=revision
> And maybe this:
> http://svn.apache.org/viewvc?revision=1155157&view=revision
> 
> -I had installed from riptano rpms.  I downloaded the src, applied the patch, and did "ant jar".  I then replaced the rpm installed cassandra jar with this new one (ugly, but I wanted to continue to run from the package).
> 
> -I think I was able to just replace the apache-cassandra-0.8.4.jar on just my jobtracker + tasktracker nodes (I need to retest from scratch to be sure, I've done a _lot_ of configuring and reconfiguring)
> 
> -Then I started getting ClassNotFound exceptions during map/reduce tasks.  Still not sure why this fix works, but the problem seems to be cloudera pig version 0.20.2+923.97-1 has two jars that match pig*.jar (which is what cassandra contrib/pig/bin/pig_cassandra uses to setup the classpath).  I had to rename /usr/lib/pig/pig.jar for things to work (leaving pig-0.8.1-cdh3u1-core.jar as the only match).
> 
> My pig script is still running, but it's the first time it didn't immediately crash.
> 
> will
> 
>