You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Jonathan Ellis <jb...@gmail.com> on 2011/07/06 20:33:29 UTC
Re: Reoganizing drivers
I don't think this is working out as well as hoped.
- the git mirror won't pick up anything under drivers/
- building the Java drivers is fragile and complicated, and there's a
lot of duplication with the "main" ant build
- patches that affect both Cassandra and JDBC are cumbersome since
they have to be committed separately (e.g.
https://issues.apache.org/jira/browse/CASSANDRA-2857)
I'm inclined to think we should move it back to trunk (but not have
multiple versions for 0.8 branch). We can still tag/branch separately
from there.
On Tue, Jun 7, 2011 at 3:08 AM, Eric Evans <ee...@rackspace.com> wrote:
>
> Sylvain and I have been discussing release issues while here at
> buzzwords, and some of the issues are related to drivers. Not
> surprising since that's a new concept for us, and there wasn't much
> thought given to the current organization.
>
> Because the CQL drivers are independently versioned and capable of
> releasing on their own timelines, the current location in SVN is
> suboptimal. There are a number of reasons why, not least of which is
> that it sets the expectation that the correct version of a driver is
> whatever corresponds to the release version of Cassandra.
>
> So, we'd like to move the drivers sub-directory up one level, making it
> look something like the following:
>
> |- branches
> |- tags
> |- site
> |- drivers
> | |- java
> | |- py
> | |- txpy
> |- trunk
>
> There are a few additional implied changes here as well, for example the
> JDBC driver will need its own build, and Cassandra's will need some
> minor changes as well (JDBC driver tests, release artifacts, etc).
>
> Does anyone object to this?
>
>
> --
> Eric Evans
> eevans@rackspace.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Reoganizing drivers
Posted by Stephen Connolly <st...@gmail.com>.
if you need a hand with m-ant-t ping me and i'll try and help. also i have
some simplifications on the deploy to maven central that i just haven't got
around to submitting a patch for (auto-retry failed upload and deploy side
artifacts at the same time... the second one will enable deploying
-SNAPSHOTs from the ci server.... which should help the drivers branch if
you want. i will be on irc tomorrow if you want to ping me... and i promise
not to point out how i think a maven based build would be easier... ;-) :-P
)
- Stephen
---
Sent from my Android phone, so random spelling mistakes, random nonsense
words and other nonsense are a direct result of using swype to type on the
screen
On 10 Jul 2011 17:56, "Eric Evans" <ee...@rackspace.com> wrote:
> On Sat, 2011-07-09 at 23:29 -0400, Rick Shaw wrote:
>> On Jul 7, 2011, at 10:53 AM, Eric Evans wrote:
>> > On Wed, 2011-07-06 at 13:33 -0500, Jonathan Ellis wrote:
>> >> - building the Java drivers is fragile and complicated, and
>> >> there's a lot of duplication with the "main" ant build
>> >
>> > Fragile how so? Because of the build-dependency on Cassandra
>> > (and/or how it is satisfied)?
>> Yes.
>
> The issues with the former are something that need to be solved either
> way. Fixing that would all but take care of the latter, but there are
> things that could be done there in the meantime.
>
> It seems like the biggest problem is that there seems to be very little
> progress here at all recently; I'll try and peel off some time in the
> next few days to help.
>
> --
> Eric Evans
> eevans@rackspace.com
>
Re: Reoganizing drivers
Posted by Eric Evans <ee...@rackspace.com>.
On Sat, 2011-07-09 at 23:29 -0400, Rick Shaw wrote:
> On Jul 7, 2011, at 10:53 AM, Eric Evans wrote:
> > On Wed, 2011-07-06 at 13:33 -0500, Jonathan Ellis wrote:
> >> - building the Java drivers is fragile and complicated, and
> >> there's a lot of duplication with the "main" ant build
> >
> > Fragile how so? Because of the build-dependency on Cassandra
> > (and/or how it is satisfied)?
> Yes.
The issues with the former are something that need to be solved either
way. Fixing that would all but take care of the latter, but there are
things that could be done there in the meantime.
It seems like the biggest problem is that there seems to be very little
progress here at all recently; I'll try and peel off some time in the
next few days to help.
--
Eric Evans
eevans@rackspace.com
Re: Reoganizing drivers
Posted by Jonathan Ellis <jb...@gmail.com>.
On Wed, Jul 13, 2011 at 10:15 AM, Eric Evans <ee...@rackspace.com> wrote:
> I don't think the story here is any different than it would be for any
> other project that depended on Cassandra like this, (i.e. it would be
> unnecessarily difficult for them as well). Of course it will never be
> as easy as treating it all like one big monolithic project, but it could
> be a whole lot easier (for anyone) and if it makes sense that they be
> treated separately (I feel strongly that it does), then I'd rather we
> fix it the right way.
>
> I realize that implicitly means that I've volunteered. :)
Works for me, then.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Reoganizing drivers
Posted by Eric Evans <ee...@rackspace.com>.
On Tue, 2011-07-12 at 16:47 -0700, Jonathan Ellis wrote:
> >> What do you mean by "but not have multiple versions for 0.8
> branch"?
>
> I mean it would live in trunk but only in trunk -- there would be no
> branches/0.8/drivers or branches/1.0/drivers.
Maybe I'm missing some svn-fu here, but how would you even do this?
Delete the directory after branching? Wouldn't merging forward to trunk
try to remove it there again?
>
> > Can't we keep the /drivers code in the trunk and just have separate
> Ant tasks for building the driver parts independent of the tasks for
> for the server?
>
> Right, this feels ideal to me. Otherwise the "right" way to handle it
> is to download a Cassandra stuff-the-driver-needs jar from the maven
> repo. I'd rather just have {cassandra} and {driver} build targets
> personally, from the same tree, rather than introducing this
> intermediate dependency.
The JDBC driver seems to be the only reason this is being brought up,
there wouldn't be much to discuss if you removed that from the equation.
None of the original reasons for moving the drivers in the first place
have changed.
I don't think the story here is any different than it would be for any
other project that depended on Cassandra like this, (i.e. it would be
unnecessarily difficult for them as well). Of course it will never be
as easy as treating it all like one big monolithic project, but it could
be a whole lot easier (for anyone) and if it makes sense that they be
treated separately (I feel strongly that it does), then I'd rather we
fix it the right way.
I realize that implicitly means that I've volunteered. :)
--
Eric Evans
eevans@rackspace.com
Re: Reoganizing drivers
Posted by Stephen Connolly <st...@gmail.com>.
On 13 July 2011 00:47, Jonathan Ellis <jb...@gmail.com> wrote:
> On Sat, Jul 9, 2011 at 8:29 PM, Rick Shaw <wf...@gmail.com> wrote:
>>
>> On Jul 7, 2011, at 10:53 AM, Eric Evans wrote:
>>
>>> On Wed, 2011-07-06 at 13:33 -0500, Jonathan Ellis wrote:
>>>> - the git mirror won't pick up anything under drivers/
>>>
>>> Has there been any effort made to have INFRA add it?
>
> Aaron asked, they said they only do standard svn trunk + branches + tags.
But I think what they mean is that you'd need to tweak things a little, e.g.
svn mkdir http://svn.apache.org/repos/asf/cassandra/server
svn mv http://svn.apache.org/repos/asf/cassandra/trunk
http://svn.apache.org/repos/asf/cassandra/tags
http://svn.apache.org/repos/asf/cassandra/branches
http://svn.apache.org/repos/asf/cassandra/server/
svn mkdir http://svn.apache.org/repos/asf/cassandra/drivers/trunk
svn mv http://svn.apache.org/repos/asf/cassandra/drivers/java
http://svn.apache.org/repos/asf/cassandra/drivers/py
http://svn.apache.org/repos/asf/cassandra/drivers/trunk/
and basically end up with at the
http://svn.apache.org/repos/asf/cassandra/ a set of directories with
trunk/tags/branches underneath.
In fact I suspect you would not even have to move the main branches
(which would be better for svn mergeinfo) as they should be able to
retain the git repo for that as is and just by structuring the drivers
directory with a trunk/tags/branches (and sure keep branches empty if
you want) they could give you
git://git.apache.org/cassandra-drivers.git for the drivers and
git://git.apache.org/cassandra.git remains in place and as is for the
core/server
if they give you any hassle just point out that they do this already
for mina, myfaces, maven, etc... and if it is a big issue you can
always rename cassandra to massandra as they seem to do it for any
project beinging with "m" ;-)
>
>>> What do you mean by "but not have multiple versions for 0.8 branch"?
>
> I mean it would live in trunk but only in trunk -- there would be no
> branches/0.8/drivers or branches/1.0/drivers.
>
>> Can't we keep the /drivers code in the trunk and just have separate Ant tasks for building the driver parts independent of the tasks for for the server?
>
> Right, this feels ideal to me. Otherwise the "right" way to handle it
> is to download a Cassandra stuff-the-driver-needs jar from the maven
> repo.
I will be providing a patch that will allow deploying -SNAPSHOTs to
the apache snapshot repository once I get some releases off the decks
at Maven, that will make life easier for the Hector guys etc (as they
can follow trunk more easily, and push -SNAPSHOTs themselves to allow
for easier user testing) but a side-effect is that it would make life
easier if you went with this approach... and remember that the maven
repo in this case does not even need to go off your machine as you can
just install the core deps in your local repo by doing a build of core
locally.
> I'd rather just have {cassandra} and {driver} build targets
> personally, from the same tree, rather than introducing this
> intermediate dependency.
It does bring up versioning... the tag for drivers releases will
include cassandra stuff too... I have the feeling that you cannot have
your cake and eat it too
>
>> Another thought would be to keep it in the separate tree as it is now, and dumb down the driver build to just build the JAR artifacts (binary, source, and javadoc). and do all testing and integration work in the server-side trunk by using just the Driver jar as a dependency for the tests in the server tree?
>
> Then you have the reverse problem, that running the test suite
> requires this other checkout to be around somewhere.
But the driver jar is supposed to be more stable, and have a much
slower release cadence. I see much less of an issue pulling the driver
jar from the (local/remote) maven repo than pulling the core stuff.
I would see this option as being closer to the way you guys seem to
want to work...
But I will raise this one question....
The tension seems to be between whether to re-release a drivers jar
with every core release, or have it with its own release cadence. To
my mind, from the Maven Pom point-of-view as long as the drivers
depend on core classes (and that includes thrift) it needs to at least
match the release cadence of core releases (i.e. you would need one
release for each release of core, and if it needs its own bug-fixes
they would be extra releases)
The major pain for releasing the drivers jar is that you have to test
it... building it is no big shakes but testing is where the effort
lies. Now the testing effort has to happen anyway at the main core
release cadence, because even if you think you don't need to rebuild
drivers for the core 0.8.2 release, you do need to test that the
drivers jar you tested against 0.8.1 still works [or at least do some
investigation to say that the changes cannot affect it working...
chaos butterflies will still want testing though*] and that is
ignoring the Maven Pom issue§
So here is the question: Does it really make sense to release the
driver jar independently? Yes you save on building that one jar, but
that is just a 2-3s increase in build time, and you cannot save on
your testing effort... the only thing that it might buy is a saving on
building effort for users of cassandra**
* and your testing should be automated and run every time so a manual
impact assessment will always be more work than just running the tests
§ which I am trying to keep out of this, as you guys seem to ignore
what I say when ever Maven comes up ;-)
** in that if they are willing to trust the Cassandra testing effort,
they could just upgrade the server and not upgrade their application
to include the matching driver.jar... but what company, that cares
about testing, is going to be happy with that story... none that I
know... upgrading the DB server... well you will want to retest your
application against the new one won't you... and testing effort should
always be greater than building effort (or else you either have no
tests or a crappy build) so it would be no skin off your nose
including an updated drivers jar.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: Reoganizing drivers
Posted by Jonathan Ellis <jb...@gmail.com>.
On Sat, Jul 9, 2011 at 8:29 PM, Rick Shaw <wf...@gmail.com> wrote:
>
> On Jul 7, 2011, at 10:53 AM, Eric Evans wrote:
>
>> On Wed, 2011-07-06 at 13:33 -0500, Jonathan Ellis wrote:
>>> - the git mirror won't pick up anything under drivers/
>>
>> Has there been any effort made to have INFRA add it?
Aaron asked, they said they only do standard svn trunk + branches + tags.
>> What do you mean by "but not have multiple versions for 0.8 branch"?
I mean it would live in trunk but only in trunk -- there would be no
branches/0.8/drivers or branches/1.0/drivers.
> Can't we keep the /drivers code in the trunk and just have separate Ant tasks for building the driver parts independent of the tasks for for the server?
Right, this feels ideal to me. Otherwise the "right" way to handle it
is to download a Cassandra stuff-the-driver-needs jar from the maven
repo. I'd rather just have {cassandra} and {driver} build targets
personally, from the same tree, rather than introducing this
intermediate dependency.
> Another thought would be to keep it in the separate tree as it is now, and dumb down the driver build to just build the JAR artifacts (binary, source, and javadoc). and do all testing and integration work in the server-side trunk by using just the Driver jar as a dependency for the tests in the server tree?
Then you have the reverse problem, that running the test suite
requires this other checkout to be around somewhere.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Reoganizing drivers
Posted by Stephen Connolly <st...@gmail.com>.
munch munch munch... this popcorn is nice... having a hard time keeping
stum... :-)
- Stephen
---
Sent from my Android phone, so random spelling mistakes, random nonsense
words and other nonsense are a direct result of using swype to type on the
screen
On 10 Jul 2011 04:30, "Rick Shaw" <wf...@gmail.com> wrote:
>
> On Jul 7, 2011, at 10:53 AM, Eric Evans wrote:
>
>> On Wed, 2011-07-06 at 13:33 -0500, Jonathan Ellis wrote:
>>> - the git mirror won't pick up anything under drivers/
>>
>> Has there been any effort made to have INFRA add it?
>> It was separated out to make it look/act like an independent project.
Cant we get an additional GIT project for drivers?
>>
>>> - building the Java drivers is fragile and complicated, and there's a
>>> lot of duplication with the "main" ant build
>>
>> Fragile how so? Because of the build-dependency on Cassandra (and/or
>> how it is satisfied)?
>
> Yes.
>>
>> What duplication are you referring to? I don't see much beyond all of
>> the boilerplate you'd see between any two ant projects.
>
>>Wow I would hardly call the current build for C* boilerplate. It is quite
comprehensive but very elaborate and complicate. A lot of clever expertise
has been applied.
>>Duplicating the necessary functionality and reaching into another existing
source tree is just complicated and error prone. Not impossible, but fragile
seems a good >description.
>>
>>> - patches that affect both Cassandra and JDBC are cumbersome since
>>> they have to be committed separately (e.g.
>>> https://issues.apache.org/jira/browse/CASSANDRA-2857)
>>
>> Well, the idea of moving it was to be able to treat it as a separate
>> project (more or less), so it follows that you'd have to independently
>> patch anything using AbstractCassandraDaemon.
>
>>This is a problem that cant be helped if they are separate projects.
RIght?
>>
>> Well, it follows that if we change an API that any project using it will
>> need to be updated as well. Since the idea behind moving the drivers
>> was to be able to treat them as separate projects, it follows that we'd
>> have to do it here as well.
>>
>>> I'm inclined to think we should move it back to trunk (but not have
>>> multiple versions for 0.8 branch). We can still tag/branch separately
>>> from there.
>>
>> What do you mean by "but not have multiple versions for 0.8 branch"?
>
>>I like the idea of it being being built as a separate project. But I also
like it being in the main trunk. Is it possible to have both?
>
>>Can't we keep the /drivers code in the trunk and just have separate Ant
tasks for building the driver parts independent of the tasks for for the
server?
>> Another thought would be to keep it in the separate tree as it is now,
and dumb down the driver build to just build the JAR artifacts (binary,
source, and javadoc). and do all testing and integration work in the
server-side trunk by using just the Driver jar as a dependency for the tests
in the server tree?
>>
>> --
>> Eric Evans
>> eevans@rackspace.com
>>
>
Re: Reoganizing drivers
Posted by Rick Shaw <wf...@gmail.com>.
On Jul 7, 2011, at 10:53 AM, Eric Evans wrote:
> On Wed, 2011-07-06 at 13:33 -0500, Jonathan Ellis wrote:
>> - the git mirror won't pick up anything under drivers/
>
> Has there been any effort made to have INFRA add it?
> It was separated out to make it look/act like an independent project. Cant we get an additional GIT project for drivers?
>
>> - building the Java drivers is fragile and complicated, and there's a
>> lot of duplication with the "main" ant build
>
> Fragile how so? Because of the build-dependency on Cassandra (and/or
> how it is satisfied)?
Yes.
>
> What duplication are you referring to? I don't see much beyond all of
> the boilerplate you'd see between any two ant projects.
>Wow I would hardly call the current build for C* boilerplate. It is quite comprehensive but very elaborate and complicate. A lot of clever expertise has been applied.
>Duplicating the necessary functionality and reaching into another existing source tree is just complicated and error prone. Not impossible, but fragile seems a good >description.
>
>> - patches that affect both Cassandra and JDBC are cumbersome since
>> they have to be committed separately (e.g.
>> https://issues.apache.org/jira/browse/CASSANDRA-2857)
>
> Well, the idea of moving it was to be able to treat it as a separate
> project (more or less), so it follows that you'd have to independently
> patch anything using AbstractCassandraDaemon.
>This is a problem that cant be helped if they are separate projects. RIght?
>
> Well, it follows that if we change an API that any project using it will
> need to be updated as well. Since the idea behind moving the drivers
> was to be able to treat them as separate projects, it follows that we'd
> have to do it here as well.
>
>> I'm inclined to think we should move it back to trunk (but not have
>> multiple versions for 0.8 branch). We can still tag/branch separately
>> from there.
>
> What do you mean by "but not have multiple versions for 0.8 branch"?
>I like the idea of it being being built as a separate project. But I also like it being in the main trunk. Is it possible to have both?
>Can't we keep the /drivers code in the trunk and just have separate Ant tasks for building the driver parts independent of the tasks for for the server?
> Another thought would be to keep it in the separate tree as it is now, and dumb down the driver build to just build the JAR artifacts (binary, source, and javadoc). and do all testing and integration work in the server-side trunk by using just the Driver jar as a dependency for the tests in the server tree?
>
> --
> Eric Evans
> eevans@rackspace.com
>
Re: Reoganizing drivers
Posted by Eric Evans <ee...@rackspace.com>.
On Wed, 2011-07-06 at 13:33 -0500, Jonathan Ellis wrote:
> - the git mirror won't pick up anything under drivers/
Has there been any effort made to have INFRA add it?
> - building the Java drivers is fragile and complicated, and there's a
> lot of duplication with the "main" ant build
Fragile how so? Because of the build-dependency on Cassandra (and/or
how it is satisfied)?
What duplication are you referring to? I don't see much beyond all of
the boilerplate you'd see between any two ant projects.
> - patches that affect both Cassandra and JDBC are cumbersome since
> they have to be committed separately (e.g.
> https://issues.apache.org/jira/browse/CASSANDRA-2857)
Well, the idea of moving it was to be able to treat it as a separate
project (more or less), so it follows that you'd have to independently
patch anything using AbstractCassandraDaemon.
Well, it follows that if we change an API that any project using it will
need to be updated as well. Since the idea behind moving the drivers
was to be able to treat them as separate projects, it follows that we'd
have to do it here as well.
> I'm inclined to think we should move it back to trunk (but not have
> multiple versions for 0.8 branch). We can still tag/branch separately
> from there.
What do you mean by "but not have multiple versions for 0.8 branch"?
--
Eric Evans
eevans@rackspace.com