You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Dylan Hutchison <dh...@mit.edu> on 2015/02/26 06:42:29 UTC
Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Hello all,
As promised
<https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
here is a design doc open for comments on implementing server-side
computation in Accumulo.
https://github.com/Accla/accumulo_stored_procedure_design
Would love to hear your opinion, especially if the proposed design pattern
matches one of *your use cases*.
Regards,
Dylan Hutchison
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Sean Busbey <bu...@cloudera.com>.
On Thu, Feb 26, 2015 at 12:59 PM, Dylan Hutchison <dh...@stevens.edu>
wrote:
> Hello Sean,
>
> The intended destination is a contrib project to Accumulo. We will first
> develop as an outside project: Graphulo <http://graphulo.mit.edu/>. We
> will create an open source Java library that implements the GraphBLAS
> functions, along with a few algorithms on top of them showing how we may
> use them for analytics. Users can use the GraphBLAS functions and
> algorithms directly, and developers can build more algorithms on top of
> them following the template we've crafted.
>
> I believe the project can be implemented without any changes to Accumulo
> core. That said, some changes to Accumulo core may make the implementation
> easier or increase performance. Looking forward to feedback.
>
> The stored procedure generalization / analogy came up as we thought through
> designs. It seems like many users could benefit from the concept.
>
> Regards,
> Dylan Hutchison
>
>
That sounds like a great contrib. Depending on how far along you already
are with the code, it might be better to start as a contrib now. (Presuming
the PMC would be in favor of it.)
Once substantial development has happened*, getting the code brought in as
a contrib will require going through the ASF IP Clearance process[1]. It's
certainly doable, but will take time.
[1]: http://incubator.apache.org/ip-clearance/
--
Sean
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Sean Busbey <bu...@cloudera.com>.
On Thu, Feb 26, 2015 at 12:59 PM, Dylan Hutchison <dh...@stevens.edu>
wrote:
> Hello Sean,
>
> The intended destination is a contrib project to Accumulo. We will first
> develop as an outside project: Graphulo <http://graphulo.mit.edu/>. We
> will create an open source Java library that implements the GraphBLAS
> functions, along with a few algorithms on top of them showing how we may
> use them for analytics. Users can use the GraphBLAS functions and
> algorithms directly, and developers can build more algorithms on top of
> them following the template we've crafted.
>
> I believe the project can be implemented without any changes to Accumulo
> core. That said, some changes to Accumulo core may make the implementation
> easier or increase performance. Looking forward to feedback.
>
> The stored procedure generalization / analogy came up as we thought through
> designs. It seems like many users could benefit from the concept.
>
> Regards,
> Dylan Hutchison
>
>
That sounds like a great contrib. Depending on how far along you already
are with the code, it might be better to start as a contrib now. (Presuming
the PMC would be in favor of it.)
Once substantial development has happened*, getting the code brought in as
a contrib will require going through the ASF IP Clearance process[1]. It's
certainly doable, but will take time.
[1]: http://incubator.apache.org/ip-clearance/
--
Sean
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Dylan Hutchison <dh...@stevens.edu>.
Hello Sean,
The intended destination is a contrib project to Accumulo. We will first
develop as an outside project: Graphulo <http://graphulo.mit.edu/>. We
will create an open source Java library that implements the GraphBLAS
functions, along with a few algorithms on top of them showing how we may
use them for analytics. Users can use the GraphBLAS functions and
algorithms directly, and developers can build more algorithms on top of
them following the template we've crafted.
I believe the project can be implemented without any changes to Accumulo
core. That said, some changes to Accumulo core may make the implementation
easier or increase performance. Looking forward to feedback.
The stored procedure generalization / analogy came up as we thought through
designs. It seems like many users could benefit from the concept.
Regards,
Dylan Hutchison
On Thu, Feb 26, 2015 at 10:13 AM, Sean Busbey <bu...@cloudera.com> wrote:
> Is the intended destination for this work within the Accumulo main project,
> within an Accumulo contrib project[1], or as a project outside of the
> Accumulo PMC?
>
> [1]: http://accumulo.apache.org/contrib.html
>
> On Wed, Feb 25, 2015 at 11:42 PM, Dylan Hutchison <dh...@mit.edu>
> wrote:
>
> > Hello all,
> >
> > As promised
> > <
> https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E
> >,
> > here is a design doc open for comments on implementing server-side
> > computation in Accumulo.
> >
> > https://github.com/Accla/accumulo_stored_procedure_design
> >
> > Would love to hear your opinion, especially if the proposed design
> pattern
> > matches one of *your use cases*.
> >
> > Regards,
> > Dylan Hutchison
> >
> >
>
>
> --
> Sean
>
--
www.cs.stevens.edu/~dhutchis
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Sean Busbey <bu...@cloudera.com>.
Is the intended destination for this work within the Accumulo main project,
within an Accumulo contrib project[1], or as a project outside of the
Accumulo PMC?
[1]: http://accumulo.apache.org/contrib.html
On Wed, Feb 25, 2015 at 11:42 PM, Dylan Hutchison <dh...@mit.edu> wrote:
> Hello all,
>
> As promised
> <https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
> here is a design doc open for comments on implementing server-side
> computation in Accumulo.
>
> https://github.com/Accla/accumulo_stored_procedure_design
>
> Would love to hear your opinion, especially if the proposed design pattern
> matches one of *your use cases*.
>
> Regards,
> Dylan Hutchison
>
>
--
Sean
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Dylan Hutchison <dh...@mit.edu>.
Hi Josh, responses follow:
Using a BatchWriter implies that you'll need some sort of resource
> management - both ensuring that the BatchWriter is close()'ed whenever a
> compaction/procedure ends and handling rejected mutations. Have you put any
> thought into how you would address these?
>
Yes, some thoughts:
- Accumulo guarantees that iterators at major compaction time are never
killed until tablet compaction completes, barring tablet server failure or
exception.
- We can close the BatchWriter in the finalize() method of the
RemoteWriteIterator. This is already done in my code for the
RemoteSourceIterator, though I don't think it matters for a Scanner. Not
ideal but maybe good enough.
- Actually, if we do all computation for a tablet from the seek() method
of the branching iterator, then we can create, use and close the
BatchWriter in a single method invocation per tablet. This way, we may
enclose all BatchWriter use in a try-finally loop, closing it in the
finally.
- This thread is relevant
<https://mail-archives.apache.org/mod_mbox/accumulo-user/201404.mbox/%3C1398567904594-9433.post@n5.nabble.com%3E>.
Note that our scenario is different in that we're operating from a major
compaction.
- ACCUMULO-1280
<https://issues.apache.org/jira/browse/ACCUMULO-1280> proposes
changing SKVI API to add a close() method, such that iterators know when
they are about to be closed and why they are about to be closed. The
iterators may then clean themselves up.
- Rejected mutations sound like something the computation should
handle. I've never encountered them personally. Any chance you could
elaborate when we need to worry about them? If it's because the remote
table has too heavy a load, then we have bigger problems. This should be
rare if the remote table to write to is a new table.
One solution I did not include this at first because its complexity may not
be necessary: *checkpoint*-and-restart-style. When an iterator dies,
Accumulo will recreate it, re-init() it and then seek() it to an
appropriate range. This range should start *right after the last entry
returned* by the iterator. If we structure the computation in such a way
that it can restart in the middle, then we can employ the following
strategy:
- Suppose the initial computation range from seek() is [a,f].
- The stored procedure table runs as normal. Eventually its
RemoteSourceIterator writes the corresponding entries for, say, [a,c]. The
RemoteSourceIterator passes c back to the BranchIterator which returns from
its seek() and prepares a top entry with row c and a top value of ""
(hasTop() is true). The computation continues when next() is called on the
BranchIterator
- Suppose Accumulo kills the iterator instead of calling next().
Accumulo re-init()s the BranchIterator which sets up the custom computation
stack again. Accumulo then seek()s with range (c,f].
- The custom computation stack may now continue with the corresponding
entries for range (c,f].
Notice that all we need is a correspondence between seek ranges and entries
computed. I still believe the above scheme is a bit too much complexity.
Suppose in the worst case we need to restart an entire tablet's worth of
computation. Okay, we have some redundant computation but *preserve
correctness*, because the result table's VersioningIterator will handle
multiple (identical) entries, in most cases.
using a wrench as a hammer -- iterators are great for performing some
> passing computation, but not really for doing some arbitrary read/writes.
> It gets back to how Accumulo/HBase comparisons where people try to compare
> Iterators and Coprocessors. They can sometimes do the same thing, but
> they're definitely different features.
>
Yes-- we are talking about different kinds of computation. That's why I
distinguish between the types of iterators in the design doc.
Traditional iterators are wrenches-- passing computation done at scan or
compaction time, not too heavy, done in a sorted, streaming manner, limited
to within a single table. They come in three flavors: one-time scan
iterators, one-time compaction iterators, and "permanent table property"
iterators set on the table at scan or compaction scope. Do correct me if I
oversimplify or if I missed a large use case.
The iterators I propose in the design doc are hammers-- heavy, one-time,
multi-table computation, that may or may not return results in sorted
order. That's why I use the phrase "stored procedure."
On *HBase Coprocessors*-- I looked up what they are here
<https://blogs.apache.org/hbase/entry/coprocessor_introduction> and it
seems our "stored procedure" idea bears many similarities to HBase
Coprocessors of the "Endpoint" type (the "Observer" type has parallels with
the Accumulo Fluo subproject). Maybe we can take some architecture lessons
from HBase. Anyone have experience they would like to share?
Regards,
Dylan Hutchison
On Thu, Feb 26, 2015 at 3:43 PM, Josh Elser <jo...@gmail.com> wrote:
> Thanks for taking the time to write this up, Dylan.
>
> I'm a little worried about the RemoteWriteIterator. Using a BatchWriter
> implies that you'll need some sort of resource management - both ensuring
> that the BatchWriter is close()'ed whenever a compaction/procedure ends and
> handling rejected mutations. Have you put any thought into how you would
> address these?
>
> I'm not familiar enough with the internals anymore, but I remember that I
> had some pains trying to write to another table during compactions when I
> was working on replication. I think as long as it's not triggered off of
> the metadata table, it wouldn't have any deadlock issues.
>
> Architecturally, it's a little worrisome, because it feels a bit like
> using a wrench as a hammer -- iterators are great for performing some
> passing computation, but not really for doing some arbitrary read/writes.
> It gets back to how Accumulo/HBase comparisons where people try to compare
> Iterators and Coprocessors. They can sometimes do the same thing, but
> they're definitely different features.
>
> Anyways, I need to stew on it some more and give it a few more reads.
> Thanks again for sharing!
>
> Dylan Hutchison wrote:
>
>> Hello all,
>>
>> As promised
>> <https://mail-archives.apache.org/mod_mbox/accumulo-user/
>> 201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%
>> 2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
>> here is a design doc open for comments on implementing server-side
>> computation in Accumulo.
>>
>> https://github.com/Accla/accumulo_stored_procedure_design
>>
>> Would love to hear your opinion, especially if the proposed design
>> pattern matches one of /your use cases/.
>>
>> Regards,
>> Dylan Hutchison
>>
>>
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Josh Elser <jo...@gmail.com>.
Dylan Hutchison wrote:
> Hi Josh, responses follow:
>
> Using a BatchWriter implies that you'll need some sort of resource
>> management - both ensuring that the BatchWriter is close()'ed whenever a
>> compaction/procedure ends and handling rejected mutations. Have you put any
>> thought into how you would address these?
>>
>
> Yes, some thoughts:
>
> - Accumulo guarantees that iterators at major compaction time are never
> killed until tablet compaction completes, barring tablet server failure or
> exception.
> - We can close the BatchWriter in the finalize() method of the
> RemoteWriteIterator. This is already done in my code for the
> RemoteSourceIterator, though I don't think it matters for a Scanner. Not
> ideal but maybe good enough.
Remember that finalize() is not guaranteed to be called by the JVM. You
are correct that you don't need to close() a Scanner (it's just good
practice).
> - Actually, if we do all computation for a tablet from the seek() method
> of the branching iterator, then we can create, use and close the
> BatchWriter in a single method invocation per tablet. This way, we may
> enclose all BatchWriter use in a try-finally loop, closing it in the
> finally.
> - This thread is relevant
> <https://mail-archives.apache.org/mod_mbox/accumulo-user/201404.mbox/%3C1398567904594-9433.post@n5.nabble.com%3E>.
> Note that our scenario is different in that we're operating from a major
> compaction.
> - ACCUMULO-1280<https://issues.apache.org/jira/browse/ACCUMULO-1280>
> proposes changing SKVI API to add a close() method, such that iterators
> know when they are about to be closed and why they are about to be closed.
> The iterators may then clean themselves up.
> One solution I did not include this at first because its complexity may not
> be necessary: *checkpoint*-and-restart-style. When an iterator dies,
> Accumulo will recreate it, re-init() it and then seek() it to an
> appropriate range. This range should start *right after the last entry
> returned* by the iterator. If we structure the computation in such a way
> that it can restart in the middle, then we can employ the following
> strategy:
>
> - Suppose the initial computation range from seek() is [a,f].
> - The stored procedure table runs as normal. Eventually its
> RemoteSourceIterator writes the corresponding entries for, say, [a,c]. The
> RemoteSourceIterator passes c back to the BranchIterator which returns from
> its seek() and prepares a top entry with row c and a top value of ""
> (hasTop() is true). The computation continues when next() is called on the
> BranchIterator
> - Suppose Accumulo kills the iterator instead of calling next().
> Accumulo re-init()s the BranchIterator which sets up the custom computation
> stack again. Accumulo then seek()s with range (c,f].
> - The custom computation stack may now continue with the corresponding
> entries for range (c,f].
>
> Notice that all we need is a correspondence between seek ranges and entries
> computed. I still believe the above scheme is a bit too much complexity.
> Suppose in the worst case we need to restart an entire tablet's worth of
> computation. Okay, we have some redundant computation but *preserve
> correctness*, because the result table's VersioningIterator will handle
> multiple (identical) entries, in most cases.
>
>
> using a wrench as a hammer -- iterators are great for performing some
>> passing computation, but not really for doing some arbitrary read/writes.
>> It gets back to how Accumulo/HBase comparisons where people try to compare
>> Iterators and Coprocessors. They can sometimes do the same thing, but
>> they're definitely different features.
>>
>
> Yes-- we are talking about different kinds of computation. That's why I
> distinguish between the types of iterators in the design doc.
>
> Traditional iterators are wrenches-- passing computation done at scan or
> compaction time, not too heavy, done in a sorted, streaming manner, limited
> to within a single table. They come in three flavors: one-time scan
> iterators, one-time compaction iterators, and "permanent table property"
> iterators set on the table at scan or compaction scope. Do correct me if I
> oversimplify or if I missed a large use case.
>
> The iterators I propose in the design doc are hammers-- heavy, one-time,
> multi-table computation, that may or may not return results in sorted
> order. That's why I use the phrase "stored procedure."
>
> On *HBase Coprocessors*-- I looked up what they are here
> <https://blogs.apache.org/hbase/entry/coprocessor_introduction> and it
> seems our "stored procedure" idea bears many similarities to HBase
> Coprocessors of the "Endpoint" type (the "Observer" type has parallels with
> the Accumulo Fluo subproject). Maybe we can take some architecture lessons
> from HBase. Anyone have experience they would like to share?
IIRC, HBase coprocessors aren't 100% as described in the Percolator
paper. I think Fluo tried to model more closely what was described, but
I don't remember anymore. Keith and Mike would know much better (since
they're building it :D).
> Regards,
> Dylan Hutchison
>
>
> On Thu, Feb 26, 2015 at 3:43 PM, Josh Elser<jo...@gmail.com> wrote:
>
>> Thanks for taking the time to write this up, Dylan.
>>
>> I'm a little worried about the RemoteWriteIterator. Using a BatchWriter
>> implies that you'll need some sort of resource management - both ensuring
>> that the BatchWriter is close()'ed whenever a compaction/procedure ends and
>> handling rejected mutations. Have you put any thought into how you would
>> address these?
>>
>> I'm not familiar enough with the internals anymore, but I remember that I
>> had some pains trying to write to another table during compactions when I
>> was working on replication. I think as long as it's not triggered off of
>> the metadata table, it wouldn't have any deadlock issues.
>>
>> Architecturally, it's a little worrisome, because it feels a bit like
>> using a wrench as a hammer -- iterators are great for performing some
>> passing computation, but not really for doing some arbitrary read/writes.
>> It gets back to how Accumulo/HBase comparisons where people try to compare
>> Iterators and Coprocessors. They can sometimes do the same thing, but
>> they're definitely different features.
>>
>> Anyways, I need to stew on it some more and give it a few more reads.
>> Thanks again for sharing!
>>
>> Dylan Hutchison wrote:
>>
>>> Hello all,
>>>
>>> As promised
>>> <https://mail-archives.apache.org/mod_mbox/accumulo-user/
>>> 201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%
>>> 2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
>>> here is a design doc open for comments on implementing server-side
>>> computation in Accumulo.
>>>
>>> https://github.com/Accla/accumulo_stored_procedure_design
>>>
>>> Would love to hear your opinion, especially if the proposed design
>>> pattern matches one of /your use cases/.
>>>
>>> Regards,
>>> Dylan Hutchison
>>>
>>>
>
>
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Dylan Hutchison <dh...@stevens.edu>.
Hi Josh, responses follow:
Using a BatchWriter implies that you'll need some sort of resource
> management - both ensuring that the BatchWriter is close()'ed whenever a
> compaction/procedure ends and handling rejected mutations. Have you put any
> thought into how you would address these?
>
Yes, some thoughts:
- Accumulo guarantees that iterators at major compaction time are never
killed until tablet compaction completes, barring tablet server failure or
exception.
- We can close the BatchWriter in the finalize() method of the
RemoteWriteIterator. This is already done in my code for the
RemoteSourceIterator, though I don't think it matters for a Scanner. Not
ideal but maybe good enough.
- Actually, if we do all computation for a tablet from the seek() method
of the branching iterator, then we can create, use and close the
BatchWriter in a single method invocation per tablet. This way, we may
enclose all BatchWriter use in a try-finally loop, closing it in the
finally.
- This thread is relevant
<https://mail-archives.apache.org/mod_mbox/accumulo-user/201404.mbox/%3C1398567904594-9433.post@n5.nabble.com%3E>.
Note that our scenario is different in that we're operating from a major
compaction.
- ACCUMULO-1280 <https://issues.apache.org/jira/browse/ACCUMULO-1280>
proposes changing SKVI API to add a close() method, such that iterators
know when they are about to be closed and why they are about to be closed.
The iterators may then clean themselves up.
One solution I did not include this at first because its complexity may not
be necessary: *checkpoint*-and-restart-style. When an iterator dies,
Accumulo will recreate it, re-init() it and then seek() it to an
appropriate range. This range should start *right after the last entry
returned* by the iterator. If we structure the computation in such a way
that it can restart in the middle, then we can employ the following
strategy:
- Suppose the initial computation range from seek() is [a,f].
- The stored procedure table runs as normal. Eventually its
RemoteSourceIterator writes the corresponding entries for, say, [a,c]. The
RemoteSourceIterator passes c back to the BranchIterator which returns from
its seek() and prepares a top entry with row c and a top value of ""
(hasTop() is true). The computation continues when next() is called on the
BranchIterator
- Suppose Accumulo kills the iterator instead of calling next().
Accumulo re-init()s the BranchIterator which sets up the custom computation
stack again. Accumulo then seek()s with range (c,f].
- The custom computation stack may now continue with the corresponding
entries for range (c,f].
Notice that all we need is a correspondence between seek ranges and entries
computed. I still believe the above scheme is a bit too much complexity.
Suppose in the worst case we need to restart an entire tablet's worth of
computation. Okay, we have some redundant computation but *preserve
correctness*, because the result table's VersioningIterator will handle
multiple (identical) entries, in most cases.
using a wrench as a hammer -- iterators are great for performing some
> passing computation, but not really for doing some arbitrary read/writes.
> It gets back to how Accumulo/HBase comparisons where people try to compare
> Iterators and Coprocessors. They can sometimes do the same thing, but
> they're definitely different features.
>
Yes-- we are talking about different kinds of computation. That's why I
distinguish between the types of iterators in the design doc.
Traditional iterators are wrenches-- passing computation done at scan or
compaction time, not too heavy, done in a sorted, streaming manner, limited
to within a single table. They come in three flavors: one-time scan
iterators, one-time compaction iterators, and "permanent table property"
iterators set on the table at scan or compaction scope. Do correct me if I
oversimplify or if I missed a large use case.
The iterators I propose in the design doc are hammers-- heavy, one-time,
multi-table computation, that may or may not return results in sorted
order. That's why I use the phrase "stored procedure."
On *HBase Coprocessors*-- I looked up what they are here
<https://blogs.apache.org/hbase/entry/coprocessor_introduction> and it
seems our "stored procedure" idea bears many similarities to HBase
Coprocessors of the "Endpoint" type (the "Observer" type has parallels with
the Accumulo Fluo subproject). Maybe we can take some architecture lessons
from HBase. Anyone have experience they would like to share?
Regards,
Dylan Hutchison
On Thu, Feb 26, 2015 at 3:43 PM, Josh Elser <jo...@gmail.com> wrote:
> Thanks for taking the time to write this up, Dylan.
>
> I'm a little worried about the RemoteWriteIterator. Using a BatchWriter
> implies that you'll need some sort of resource management - both ensuring
> that the BatchWriter is close()'ed whenever a compaction/procedure ends and
> handling rejected mutations. Have you put any thought into how you would
> address these?
>
> I'm not familiar enough with the internals anymore, but I remember that I
> had some pains trying to write to another table during compactions when I
> was working on replication. I think as long as it's not triggered off of
> the metadata table, it wouldn't have any deadlock issues.
>
> Architecturally, it's a little worrisome, because it feels a bit like
> using a wrench as a hammer -- iterators are great for performing some
> passing computation, but not really for doing some arbitrary read/writes.
> It gets back to how Accumulo/HBase comparisons where people try to compare
> Iterators and Coprocessors. They can sometimes do the same thing, but
> they're definitely different features.
>
> Anyways, I need to stew on it some more and give it a few more reads.
> Thanks again for sharing!
>
> Dylan Hutchison wrote:
>
>> Hello all,
>>
>> As promised
>> <https://mail-archives.apache.org/mod_mbox/accumulo-user/
>> 201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%
>> 2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
>> here is a design doc open for comments on implementing server-side
>> computation in Accumulo.
>>
>> https://github.com/Accla/accumulo_stored_procedure_design
>>
>> Would love to hear your opinion, especially if the proposed design
>> pattern matches one of /your use cases/.
>>
>> Regards,
>> Dylan Hutchison
>>
>>
--
www.cs.stevens.edu/~dhutchis
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Dylan Hutchison <dh...@mit.edu>.
Hi Josh, responses follow:
Using a BatchWriter implies that you'll need some sort of resource
> management - both ensuring that the BatchWriter is close()'ed whenever a
> compaction/procedure ends and handling rejected mutations. Have you put any
> thought into how you would address these?
>
Yes, some thoughts:
- Accumulo guarantees that iterators at major compaction time are never
killed until tablet compaction completes, barring tablet server failure or
exception.
- We can close the BatchWriter in the finalize() method of the
RemoteWriteIterator. This is already done in my code for the
RemoteSourceIterator, though I don't think it matters for a Scanner. Not
ideal but maybe good enough.
- Actually, if we do all computation for a tablet from the seek() method
of the branching iterator, then we can create, use and close the
BatchWriter in a single method invocation per tablet. This way, we may
enclose all BatchWriter use in a try-finally loop, closing it in the
finally.
- This thread is relevant
<https://mail-archives.apache.org/mod_mbox/accumulo-user/201404.mbox/%3C1398567904594-9433.post@n5.nabble.com%3E>.
Note that our scenario is different in that we're operating from a major
compaction.
- ACCUMULO-1280
<https://issues.apache.org/jira/browse/ACCUMULO-1280> proposes
changing SKVI API to add a close() method, such that iterators know when
they are about to be closed and why they are about to be closed. The
iterators may then clean themselves up.
- Rejected mutations sound like something the computation should
handle. I've never encountered them personally. Any chance you could
elaborate when we need to worry about them? If it's because the remote
table has too heavy a load, then we have bigger problems. This should be
rare if the remote table to write to is a new table.
One solution I did not include this at first because its complexity may not
be necessary: *checkpoint*-and-restart-style. When an iterator dies,
Accumulo will recreate it, re-init() it and then seek() it to an
appropriate range. This range should start *right after the last entry
returned* by the iterator. If we structure the computation in such a way
that it can restart in the middle, then we can employ the following
strategy:
- Suppose the initial computation range from seek() is [a,f].
- The stored procedure table runs as normal. Eventually its
RemoteSourceIterator writes the corresponding entries for, say, [a,c]. The
RemoteSourceIterator passes c back to the BranchIterator which returns from
its seek() and prepares a top entry with row c and a top value of ""
(hasTop() is true). The computation continues when next() is called on the
BranchIterator
- Suppose Accumulo kills the iterator instead of calling next().
Accumulo re-init()s the BranchIterator which sets up the custom computation
stack again. Accumulo then seek()s with range (c,f].
- The custom computation stack may now continue with the corresponding
entries for range (c,f].
Notice that all we need is a correspondence between seek ranges and entries
computed. I still believe the above scheme is a bit too much complexity.
Suppose in the worst case we need to restart an entire tablet's worth of
computation. Okay, we have some redundant computation but *preserve
correctness*, because the result table's VersioningIterator will handle
multiple (identical) entries, in most cases.
using a wrench as a hammer -- iterators are great for performing some
> passing computation, but not really for doing some arbitrary read/writes.
> It gets back to how Accumulo/HBase comparisons where people try to compare
> Iterators and Coprocessors. They can sometimes do the same thing, but
> they're definitely different features.
>
Yes-- we are talking about different kinds of computation. That's why I
distinguish between the types of iterators in the design doc.
Traditional iterators are wrenches-- passing computation done at scan or
compaction time, not too heavy, done in a sorted, streaming manner, limited
to within a single table. They come in three flavors: one-time scan
iterators, one-time compaction iterators, and "permanent table property"
iterators set on the table at scan or compaction scope. Do correct me if I
oversimplify or if I missed a large use case.
The iterators I propose in the design doc are hammers-- heavy, one-time,
multi-table computation, that may or may not return results in sorted
order. That's why I use the phrase "stored procedure."
On *HBase Coprocessors*-- I looked up what they are here
<https://blogs.apache.org/hbase/entry/coprocessor_introduction> and it
seems our "stored procedure" idea bears many similarities to HBase
Coprocessors of the "Endpoint" type (the "Observer" type has parallels with
the Accumulo Fluo subproject). Maybe we can take some architecture lessons
from HBase. Anyone have experience they would like to share?
Regards,
Dylan Hutchison
On Thu, Feb 26, 2015 at 3:43 PM, Josh Elser <jo...@gmail.com> wrote:
> Thanks for taking the time to write this up, Dylan.
>
> I'm a little worried about the RemoteWriteIterator. Using a BatchWriter
> implies that you'll need some sort of resource management - both ensuring
> that the BatchWriter is close()'ed whenever a compaction/procedure ends and
> handling rejected mutations. Have you put any thought into how you would
> address these?
>
> I'm not familiar enough with the internals anymore, but I remember that I
> had some pains trying to write to another table during compactions when I
> was working on replication. I think as long as it's not triggered off of
> the metadata table, it wouldn't have any deadlock issues.
>
> Architecturally, it's a little worrisome, because it feels a bit like
> using a wrench as a hammer -- iterators are great for performing some
> passing computation, but not really for doing some arbitrary read/writes.
> It gets back to how Accumulo/HBase comparisons where people try to compare
> Iterators and Coprocessors. They can sometimes do the same thing, but
> they're definitely different features.
>
> Anyways, I need to stew on it some more and give it a few more reads.
> Thanks again for sharing!
>
> Dylan Hutchison wrote:
>
>> Hello all,
>>
>> As promised
>> <https://mail-archives.apache.org/mod_mbox/accumulo-user/
>> 201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%
>> 2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
>> here is a design doc open for comments on implementing server-side
>> computation in Accumulo.
>>
>> https://github.com/Accla/accumulo_stored_procedure_design
>>
>> Would love to hear your opinion, especially if the proposed design
>> pattern matches one of /your use cases/.
>>
>> Regards,
>> Dylan Hutchison
>>
>>
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Josh Elser <jo...@gmail.com>.
Thanks for taking the time to write this up, Dylan.
I'm a little worried about the RemoteWriteIterator. Using a BatchWriter
implies that you'll need some sort of resource management - both
ensuring that the BatchWriter is close()'ed whenever a
compaction/procedure ends and handling rejected mutations. Have you put
any thought into how you would address these?
I'm not familiar enough with the internals anymore, but I remember that
I had some pains trying to write to another table during compactions
when I was working on replication. I think as long as it's not triggered
off of the metadata table, it wouldn't have any deadlock issues.
Architecturally, it's a little worrisome, because it feels a bit like
using a wrench as a hammer -- iterators are great for performing some
passing computation, but not really for doing some arbitrary
read/writes. It gets back to how Accumulo/HBase comparisons where people
try to compare Iterators and Coprocessors. They can sometimes do the
same thing, but they're definitely different features.
Anyways, I need to stew on it some more and give it a few more reads.
Thanks again for sharing!
Dylan Hutchison wrote:
> Hello all,
>
> As promised
> <https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
> here is a design doc open for comments on implementing server-side
> computation in Accumulo.
>
> https://github.com/Accla/accumulo_stored_procedure_design
>
> Would love to hear your opinion, especially if the proposed design
> pattern matches one of /your use cases/.
>
> Regards,
> Dylan Hutchison
>
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Josh Elser <jo...@gmail.com>.
Thanks for taking the time to write this up, Dylan.
I'm a little worried about the RemoteWriteIterator. Using a BatchWriter
implies that you'll need some sort of resource management - both
ensuring that the BatchWriter is close()'ed whenever a
compaction/procedure ends and handling rejected mutations. Have you put
any thought into how you would address these?
I'm not familiar enough with the internals anymore, but I remember that
I had some pains trying to write to another table during compactions
when I was working on replication. I think as long as it's not triggered
off of the metadata table, it wouldn't have any deadlock issues.
Architecturally, it's a little worrisome, because it feels a bit like
using a wrench as a hammer -- iterators are great for performing some
passing computation, but not really for doing some arbitrary
read/writes. It gets back to how Accumulo/HBase comparisons where people
try to compare Iterators and Coprocessors. They can sometimes do the
same thing, but they're definitely different features.
Anyways, I need to stew on it some more and give it a few more reads.
Thanks again for sharing!
Dylan Hutchison wrote:
> Hello all,
>
> As promised
> <https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
> here is a design doc open for comments on implementing server-side
> computation in Accumulo.
>
> https://github.com/Accla/accumulo_stored_procedure_design
>
> Would love to hear your opinion, especially if the proposed design
> pattern matches one of /your use cases/.
>
> Regards,
> Dylan Hutchison
>
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Dylan Hutchison <dh...@mit.edu>.
Hi Christopher, responses after yours:
It's a clever way to leverage existing Accumulo behaviors (full major
> compaction) to act as clients in order to perform a parallel operation to
> populate a new table. Have you tried this method in practice at all, yet?
> What pitfalls have you run into, perhaps regarding client-side static state
> in the JVM, or resource management issues within the tablet servers?
So far I have had success implementing a variant of the design doc:
everything except the RemoteWriteIterator. I ran two RemoteSourceIterator
<https://github.com/Accla/d4m_api_java/blob/master/src/main/java/edu/mit/ll/graphulo/RemoteSourceIterator.java>s
and a DotMultIterator
<https://github.com/Accla/d4m_api_java/blob/master/src/main/java/edu/mit/ll/graphulo/DotRemoteSourceIterator.java>
on a new Accumulo table via a one-time manual major compaction. It works
as expected. Test code is here for RemoteSource
<https://github.com/Accla/d4m_api_java/blob/master/src/test/java/edu/mit/ll/graphulo/RemoteIteratorTest.java>
and here for the Dot
<https://github.com/Accla/d4m_api_java/blob/master/src/test/java/edu/mit/ll/graphulo/DotRemoteIteratorTest.java>.
That said, the tests are a very small size. I expect unforeseen issues,
maybe the ones you describe, will arise when we scale up.
Later this week I will prototype the design doc as written, using a stored
procedure table with table splits and iterators on major compaction and
does *not* store any data.
What do you think the similarities and differences are with other parallel
> execution methods that one could use to achieve the same results (like
> Map/Reduce)?
Similar to Yarn, MapReduce is great if we wanted to run computations over
an entire table, i.e. if we valued high throughput over low latency. The
design doc addresses the reverse use case, when we want to select subsets
of a table and value low latency over high throughput. This is where
Accumulo lends us strength. There is also a chance we can stream results
if our computation preserves sorted order-- see the temporary table section
<https://github.com/Accla/accumulo_stored_procedure_design#temporary-tables>
.
Also, do you have any code available for an example RemoteSourceIterator,
> which we might be able to try? The Transpose one seemed simple enough, but
> any others would be neat to try, also.
See above and the links in the design doc. Here is a RemoteMergeIterator
<https://github.com/Accla/d4m_api_java/blob/master/src/main/java/edu/mit/ll/graphulo/RemoteMergeIterator.java>.
There is some mess in the code as I shifted between designs.
Do you have any thoughts on whether there should be some abstract base
> class available in Accumulo (vs. as part of the contrib) to support these
> iterators and handle the boiler-plate stuff of setting up/serializing the
> client configuration when the procedure executes, or utilities to help
> create a stored procedure table?
Not sure yet. I can report back to the Accumulo community after one pass
implementing the design. For now, I'd like to see the Accumulo community's
opinion on the design and merit.
The temporary table concept may be worth Accumulo core consideration.
I wonder if one result of this project is writing a guideline /
best-practices document on *where to place iterators, *or in other words,
where to place computation.
Regards,
Dylan Hutchison
On Thu, Feb 26, 2015 at 2:34 PM, Christopher <ct...@apache.org> wrote:
> Hi Dylan,
>
> It's a clever way to leverage existing Accumulo behaviors (full major
> compaction) to act as clients in order to perform a parallel operation to
> populate a new table. Have you tried this method in practice at all, yet?
> What pitfalls have you run into, perhaps regarding client-side static state
> in the JVM, or resource management issues within the tablet servers? What
> do you think the similarities and differences are with other parallel
> execution methods that one could use to achieve the same results (like
> Map/Reduce)?
>
> Also, do you have any code available for an example RemoteSourceIterator,
> which we might be able to try? The Transpose one seemed simple enough, but
> any others would be neat to try, also.
>
> Do you have any thoughts on whether there should be some abstract base
> class available in Accumulo (vs. as part of the contrib) to support these
> iterators and handle the boiler-plate stuff of setting up/serializing the
> client configuration when the procedure executes, or utilities to help
> create a stored procedure table?
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Thu, Feb 26, 2015 at 12:42 AM, Dylan Hutchison <dh...@mit.edu>
> wrote:
>
>> Hello all,
>>
>> As promised
>> <https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
>> here is a design doc open for comments on implementing server-side
>> computation in Accumulo.
>>
>> https://github.com/Accla/accumulo_stored_procedure_design
>>
>> Would love to hear your opinion, especially if the proposed design
>> pattern matches one of *your use cases*.
>>
>> Regards,
>> Dylan Hutchison
>>
>>
>
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Dylan Hutchison <dh...@mit.edu>.
Hi Christopher, responses after yours:
It's a clever way to leverage existing Accumulo behaviors (full major
> compaction) to act as clients in order to perform a parallel operation to
> populate a new table. Have you tried this method in practice at all, yet?
> What pitfalls have you run into, perhaps regarding client-side static state
> in the JVM, or resource management issues within the tablet servers?
So far I have had success implementing a variant of the design doc:
everything except the RemoteWriteIterator. I ran two RemoteSourceIterator
<https://github.com/Accla/d4m_api_java/blob/master/src/main/java/edu/mit/ll/graphulo/RemoteSourceIterator.java>s
and a DotMultIterator
<https://github.com/Accla/d4m_api_java/blob/master/src/main/java/edu/mit/ll/graphulo/DotRemoteSourceIterator.java>
on a new Accumulo table via a one-time manual major compaction. It works
as expected. Test code is here for RemoteSource
<https://github.com/Accla/d4m_api_java/blob/master/src/test/java/edu/mit/ll/graphulo/RemoteIteratorTest.java>
and here for the Dot
<https://github.com/Accla/d4m_api_java/blob/master/src/test/java/edu/mit/ll/graphulo/DotRemoteIteratorTest.java>.
That said, the tests are a very small size. I expect unforeseen issues,
maybe the ones you describe, will arise when we scale up.
Later this week I will prototype the design doc as written, using a stored
procedure table with table splits and iterators on major compaction and
does *not* store any data.
What do you think the similarities and differences are with other parallel
> execution methods that one could use to achieve the same results (like
> Map/Reduce)?
Similar to Yarn, MapReduce is great if we wanted to run computations over
an entire table, i.e. if we valued high throughput over low latency. The
design doc addresses the reverse use case, when we want to select subsets
of a table and value low latency over high throughput. This is where
Accumulo lends us strength. There is also a chance we can stream results
if our computation preserves sorted order-- see the temporary table section
<https://github.com/Accla/accumulo_stored_procedure_design#temporary-tables>
.
Also, do you have any code available for an example RemoteSourceIterator,
> which we might be able to try? The Transpose one seemed simple enough, but
> any others would be neat to try, also.
See above and the links in the design doc. Here is a RemoteMergeIterator
<https://github.com/Accla/d4m_api_java/blob/master/src/main/java/edu/mit/ll/graphulo/RemoteMergeIterator.java>.
There is some mess in the code as I shifted between designs.
Do you have any thoughts on whether there should be some abstract base
> class available in Accumulo (vs. as part of the contrib) to support these
> iterators and handle the boiler-plate stuff of setting up/serializing the
> client configuration when the procedure executes, or utilities to help
> create a stored procedure table?
Not sure yet. I can report back to the Accumulo community after one pass
implementing the design. For now, I'd like to see the Accumulo community's
opinion on the design and merit.
The temporary table concept may be worth Accumulo core consideration.
I wonder if one result of this project is writing a guideline /
best-practices document on *where to place iterators, *or in other words,
where to place computation.
Regards,
Dylan Hutchison
On Thu, Feb 26, 2015 at 2:34 PM, Christopher <ct...@apache.org> wrote:
> Hi Dylan,
>
> It's a clever way to leverage existing Accumulo behaviors (full major
> compaction) to act as clients in order to perform a parallel operation to
> populate a new table. Have you tried this method in practice at all, yet?
> What pitfalls have you run into, perhaps regarding client-side static state
> in the JVM, or resource management issues within the tablet servers? What
> do you think the similarities and differences are with other parallel
> execution methods that one could use to achieve the same results (like
> Map/Reduce)?
>
> Also, do you have any code available for an example RemoteSourceIterator,
> which we might be able to try? The Transpose one seemed simple enough, but
> any others would be neat to try, also.
>
> Do you have any thoughts on whether there should be some abstract base
> class available in Accumulo (vs. as part of the contrib) to support these
> iterators and handle the boiler-plate stuff of setting up/serializing the
> client configuration when the procedure executes, or utilities to help
> create a stored procedure table?
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Thu, Feb 26, 2015 at 12:42 AM, Dylan Hutchison <dh...@mit.edu>
> wrote:
>
>> Hello all,
>>
>> As promised
>> <https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
>> here is a design doc open for comments on implementing server-side
>> computation in Accumulo.
>>
>> https://github.com/Accla/accumulo_stored_procedure_design
>>
>> Would love to hear your opinion, especially if the proposed design
>> pattern matches one of *your use cases*.
>>
>> Regards,
>> Dylan Hutchison
>>
>>
>
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Christopher <ct...@apache.org>.
Hi Dylan,
It's a clever way to leverage existing Accumulo behaviors (full major
compaction) to act as clients in order to perform a parallel operation to
populate a new table. Have you tried this method in practice at all, yet?
What pitfalls have you run into, perhaps regarding client-side static state
in the JVM, or resource management issues within the tablet servers? What
do you think the similarities and differences are with other parallel
execution methods that one could use to achieve the same results (like
Map/Reduce)?
Also, do you have any code available for an example RemoteSourceIterator,
which we might be able to try? The Transpose one seemed simple enough, but
any others would be neat to try, also.
Do you have any thoughts on whether there should be some abstract base
class available in Accumulo (vs. as part of the contrib) to support these
iterators and handle the boiler-plate stuff of setting up/serializing the
client configuration when the procedure executes, or utilities to help
create a stored procedure table?
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Thu, Feb 26, 2015 at 12:42 AM, Dylan Hutchison <dh...@mit.edu> wrote:
> Hello all,
>
> As promised
> <https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
> here is a design doc open for comments on implementing server-side
> computation in Accumulo.
>
> https://github.com/Accla/accumulo_stored_procedure_design
>
> Would love to hear your opinion, especially if the proposed design pattern
> matches one of *your use cases*.
>
> Regards,
> Dylan Hutchison
>
>
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Christopher <ct...@apache.org>.
Hi Dylan,
It's a clever way to leverage existing Accumulo behaviors (full major
compaction) to act as clients in order to perform a parallel operation to
populate a new table. Have you tried this method in practice at all, yet?
What pitfalls have you run into, perhaps regarding client-side static state
in the JVM, or resource management issues within the tablet servers? What
do you think the similarities and differences are with other parallel
execution methods that one could use to achieve the same results (like
Map/Reduce)?
Also, do you have any code available for an example RemoteSourceIterator,
which we might be able to try? The Transpose one seemed simple enough, but
any others would be neat to try, also.
Do you have any thoughts on whether there should be some abstract base
class available in Accumulo (vs. as part of the contrib) to support these
iterators and handle the boiler-plate stuff of setting up/serializing the
client configuration when the procedure executes, or utilities to help
create a stored procedure table?
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Thu, Feb 26, 2015 at 12:42 AM, Dylan Hutchison <dh...@mit.edu> wrote:
> Hello all,
>
> As promised
> <https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
> here is a design doc open for comments on implementing server-side
> computation in Accumulo.
>
> https://github.com/Accla/accumulo_stored_procedure_design
>
> Would love to hear your opinion, especially if the proposed design pattern
> matches one of *your use cases*.
>
> Regards,
> Dylan Hutchison
>
>
Re: Design-for-comment: Accumulo Server-Side Computation: Stored
Procedure Tables starring SpGEMM
Posted by Sean Busbey <bu...@cloudera.com>.
Is the intended destination for this work within the Accumulo main project,
within an Accumulo contrib project[1], or as a project outside of the
Accumulo PMC?
[1]: http://accumulo.apache.org/contrib.html
On Wed, Feb 25, 2015 at 11:42 PM, Dylan Hutchison <dh...@mit.edu> wrote:
> Hello all,
>
> As promised
> <https://mail-archives.apache.org/mod_mbox/accumulo-user/201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
> here is a design doc open for comments on implementing server-side
> computation in Accumulo.
>
> https://github.com/Accla/accumulo_stored_procedure_design
>
> Would love to hear your opinion, especially if the proposed design pattern
> matches one of *your use cases*.
>
> Regards,
> Dylan Hutchison
>
>
--
Sean