idea for performance improvement with alignFormat

[Eric Owhadi <er...@esgyn.com>]

Good morning Trafodioneers,



Align format is great from a storage, write and “sometime” read performance.

The “sometime”, is what I think we could address.

In alignFormat, if you do a select *, then yes, the performance will be
better than regular format.

But if you do a select a, on a wide row with many columns, alignFormat will
bring back the full set of columns, like if it was a select *, when regular
format would cherry pick.



We could implement an HBase custom filter, using the transformCell() step
to create a returned row in a new custom format (like aligned format but
with only the requested columns), aware of alignFormat, that will pick
inside an AlignFormat cell, only the column values requested.



Here is the description of transformCell step of the Custom Filter
framework.



transformCell() :

Once the cell has passed the check and is available, the transform call
allows the filter to modify the cell, before it is added to the resulting
row.



Thought?

Eric

RE: idea for performance improvement with alignFormat

[Dave Birdsall <da...@esgyn.com>]

So, we do the "project" in the RegionServer...  Sounds like a winner. Go for
it!

This brings me back to the point we discussed in the MDAM thread though. An
Executor-in-RegionServer would be able to do this too.

I'm wondering if we can build an Executor-in-RegionServer architecture on
top of the various filters and coprocessors available in the RegionServer
today.

Well, really, the question I'm asking is: Is the current direction of
filters and coprocessors a series of one-offs or do they fit together in a
coherent way that we can make as powerful as the former Executor-in-DP2
architecture of SQL/MX?

-----Original Message-----
From: Eric Owhadi [mailto:eric.owhadi@esgyn.com]
Sent: Tuesday, March 29, 2016 8:57 AM
To: dev@trafodion.incubator.apache.org
Subject: idea for performance improvement with alignFormat

Good morning Trafodioneers,



Align format is great from a storage, write and “sometime” read performance.

The “sometime”, is what I think we could address.

In alignFormat, if you do a select *, then yes, the performance will be
better than regular format.

But if you do a select a, on a wide row with many columns, alignFormat will
bring back the full set of columns, like if it was a select *, when regular
format would cherry pick.



We could implement an HBase custom filter, using the transformCell() step to
create a returned row in a new custom format (like aligned format but with
only the requested columns), aware of alignFormat, that will pick inside an
AlignFormat cell, only the column values requested.



Here is the description of transformCell step of the Custom Filter
framework.



transformCell() :

Once the cell has passed the check and is available, the transform call
allows the filter to modify the cell, before it is added to the resulting
row.



Thought?

Eric