In memory code and query executions

[Corry Opdenakker <co...@bestdata.be>]

Hi all,

Is it possible to execute queries towards an embedded cassandra db whyle
bypassing completely the TCP (or IPC) protocol stack?
Apparantly the embedded cassandra is by default accessed using localhost as
hostname which will result in an IPC optimized connection I assume.
Is there a way to fully omit the Tcp/ipc stack and execute queries directly
in-memory at the cassandra database? preferrably in a (query resultset ->
to -> appcode) zero-copy approach.

Cheers, C.

Re: In memory code and query executions

[Jonathan Haddad <jo...@jonhaddad.com>]

Agreed with Nate.  This is generally one of those "if you have to ask how
it's done, you shouldn't be doing it" ideas.

To add to his points above, deploying new versions of you app with this
model is an operational nightmare.  Now you've tightly coupled new versions
of your app to doing a full cluster restart.  You're also not going to find
many people who have done this to help you out when you run into problems.
Every issue you hit, when you ask on the ML, will likely result in "well
you're doing something unsupported so good luck, have fun, but we can't
help you".

On Wed, May 4, 2016 at 11:16 AM Nate McCall <na...@thelastpickle.com> wrote:

> On Mon, May 2, 2016 at 11:04 AM, Corry Opdenakker <co...@bestdata.be>
> wrote:
>
>> Hi all,
>>
>> Is it possible to execute queries towards an embedded cassandra db whyle
>> bypassing completely the TCP (or IPC) protocol stack?
>>
>
> tl,dr: it is not for the faint of heart and you must understand *exactly*
> what you are doing.
>
> First I have to ask is there something specific that is not working the
> way you anticipate?
>
> Short answer is yes, though:
> https://github.com/apache/cassandra/tree/cassandra-2.1/examples/client_only
>
> This was removed in > 2.1 because very few people were using it and it was
> confusing to have there as an "example."
>
> I would not call this embedded so much as running a "client-mode proxy"
> but same idea.
>
> Apparantly the embedded cassandra is by default accessed using localhost
>> as hostname which will result in an IPC optimized connection I assume.
>>
>
> Not quite sure what you mean here?
>
>
>> Is there a way to fully omit the Tcp/ipc stack and execute queries
>> directly in-memory at the cassandra database? preferrably in a (query
>> resultset -> to -> appcode) zero-copy approach.
>>
>>
> Again, yes per the link above, but you would need to modify a few things
> for recent versions. The general approach is there however. You could even
> go a level below QueryProcessor and invoke methods on StorageProxy
> directly, bypassing the parse/PS lookup.
>
> That all said, you need to understand:
> - These are all internal APIs and as such can and will change
> substantially without warning even between point releases
> - Understanding the internals to use them correctly at this level requires
> a deep understanding of the code base
> - You will be bypassing a substantial amount of validation and could
> easily insert data that will corrupt your table
> - You can potentially put a lot more pressure on portions of the system
> that anticipate upstream throttling
>
> In sum: it's possible, but put something in production first using
> standard APIs before you go this deep. This is not the level at which you
> want to write your first app against Cassandra.
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> CTO
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: In memory code and query executions

[Nate McCall <na...@thelastpickle.com>]

On Mon, May 2, 2016 at 11:04 AM, Corry Opdenakker <co...@bestdata.be> wrote:

> Hi all,
>
> Is it possible to execute queries towards an embedded cassandra db whyle
> bypassing completely the TCP (or IPC) protocol stack?
>

tl,dr: it is not for the faint of heart and you must understand *exactly*
what you are doing.

First I have to ask is there something specific that is not working the way
you anticipate?

Short answer is yes, though:
https://github.com/apache/cassandra/tree/cassandra-2.1/examples/client_only

This was removed in > 2.1 because very few people were using it and it was
confusing to have there as an "example."

I would not call this embedded so much as running a "client-mode proxy" but
same idea.

Apparantly the embedded cassandra is by default accessed using localhost as
> hostname which will result in an IPC optimized connection I assume.
>

Not quite sure what you mean here?


> Is there a way to fully omit the Tcp/ipc stack and execute queries
> directly in-memory at the cassandra database? preferrably in a (query
> resultset -> to -> appcode) zero-copy approach.
>
>
Again, yes per the link above, but you would need to modify a few things
for recent versions. The general approach is there however. You could even
go a level below QueryProcessor and invoke methods on StorageProxy
directly, bypassing the parse/PS lookup.

That all said, you need to understand:
- These are all internal APIs and as such can and will change substantially
without warning even between point releases
- Understanding the internals to use them correctly at this level requires
a deep understanding of the code base
- You will be bypassing a substantial amount of validation and could easily
insert data that will corrupt your table
- You can potentially put a lot more pressure on portions of the system
that anticipate upstream throttling

In sum: it's possible, but put something in production first using standard
APIs before you go this deep. This is not the level at which you want to
write your first app against Cassandra.


-- 
-----------------
Nate McCall
Austin, TX
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com