using stateful iterators to do filtering during major compaction?

[Jonathan LASKO <jo...@raytheon.com>]

Hi Accumulo wizards,

TL;DR - this is a question about custom iterators and saving state (or seeking backwards) in order to filter / mask data during major compaction.

For a project I'm working on, we would like to be able to use one entry to filter other entries in the same row. (I will call the first entry the 'filtering key.') To do this, we would ensure that this 'filtering key' lexicographically precedes the other entries it would be used on.

There is, of course, a "snag" with this idea: the iterator could simply read and save in memory the entry and then use it for subsequent filtering, were it not for the fact that the iterator stack can be dropped and re-initialized at any point in the row, including cf's/cq's that are already past the 'filtering key.' Our understanding is that the tserver processes can (and do!) restart and re-initialize the iterator stack at any point. When this happens, the tserver will "seek(...)" the newly re-initialized iterator stack back to the same row/cf/cq that the previous incarnation of the stack was on when it got re-initialized.

When this teardown/re-init happens, the tserver doesn't call deepCopy(...) on the iterator stack; it just calls init(...). (At least, this is our experience in Accumulo 1.6.2.) For this reason, it is seen as a risky proposition to try to keep state in the iterators. (Josh Elser acknowledges this in his presentation on designing and testing custom iterators for Accumulo, https://www.slideshare.net/je2451/designing-and-testing-accumulo-iterators).

Nevertheless, for the scantime scope, I believe we can use WholeRowIterator to ensure that we don't ever return data for a row until we've read the entire row, thus avoiding the need to keep state in the iterators. (If the iterator stack gets re-initialized, we should start over from the beginning of the row.)

Our problem comes when we want to use this filter in majc.compaction scope to actually filter the masked data out of the system entirely. In this case, the WholeRowIterator approach wouldn't seem to be usable (because Accumulo only allows us to set filters for compaction time but not iterators).

Here are our questions:

(1) Has Accumulo's behavior when tearing down and re-initializing an iterator stack changed between 1.6.2 and the latest version? (I.e. is deepCopy now called?)

(2) Are there any other ways in which storing state across iterator stack teardowns has been made any easier?

(3) If not, are there any other tricks/hacks which we might consider using (albeit with caution) to store state or otherwise accomplish this? (Options we've mused about include figuring out another way for the iterators to store state beyond themselves -- can iterators write to the IteratorEnvironment to influence future iterator instantiations? -- and/or allowing the iterators to seek backwards to get the 'filtering key' they need.)

(4) Also: any downsides to using the WholeRowIterator we should keep in mind?

Thanks in advance,

Jonathan

Re: using stateful iterators to do filtering during major compaction?

[Josh Elser <jo...@gmail.com>]

Hi Jonathan -- good questions. I've tried to address them inline.

Jonathan LASKO wrote:
> Hi Accumulo wizards,
>
> TL;DR - this is a question about custom iterators and saving state (or seeking backwards) in order to filter / mask data during major compaction.
>
> For a project I'm working on, we would like to be able to use one entry to filter other entries in the same row. (I will call the first entry the 'filtering key.') To do this, we would ensure that this 'filtering key' lexicographically precedes the other entries it would be used on.
>
> There is, of course, a "snag" with this idea: the iterator could simply read and save in memory the entry and then use it for subsequent filtering, were it not for the fact that the iterator stack can be dropped and re-initialized at any point in the row, including cf's/cq's that are already past the 'filtering key.' Our understanding is that the tserver processes can (and do!) restart and re-initialize the iterator stack at any point. When this happens, the tserver will "seek(...)" the newly re-initialized iterator stack back to the same row/cf/cq that the previous incarnation of the stack was on when it got re-initialized.

This teardown/re-seek will only happen after your iterator emits a 
Key-Value pair (really, this is when your entire "stack" emits a 
Key-Value pair to the caller, but developing iterators with the 
knowledge of their operating context is bad design).

This is a long-winded of saying that if you can safely buffer an entire 
row, you can rest assured that Accumulo will not tear you down 
mid-operation. The reason this is not advertised is because it is 
caveated by the size of your rows. It's a pretty common gotcha for user 
data to not be constrained in size (99.9% of rows are small, but a .1% 
are huge).

> When this teardown/re-init happens, the tserver doesn't call deepCopy(...) on the iterator stack; it just calls init(...). (At least, this is our experience in Accumulo 1.6.2.) For this reason, it is seen as a risky proposition to try to keep state in the iterators. (Josh Elser acknowledges this in his presentation on designing and testing custom iterators for Accumulo, https://www.slideshare.net/je2451/designing-and-testing-accumulo-iterators).

Yeah, I've (re?)noticed this one myself recently (working on 
ACCUMULO-3208). Really, the only context you're guaranteed is a 
start-key in the Range you're passed via seek() which would prevent your 
iterator from returning duplicate data.

I'm not sure if it would be simple for us to hold on to the previous 
instance in the TabletServer memory or not. I would have to do a fairly 
deep dive into the code to see if there's something more we could do here.

> Nevertheless, for the scantime scope, I believe we can use WholeRowIterator to ensure that we don't ever return data for a row until we've read the entire row, thus avoiding the need to keep state in the iterators. (If the iterator stack gets re-initialized, we should start over from the beginning of the row.)
>
> Our problem comes when we want to use this filter in majc.compaction scope to actually filter the masked data out of the system entirely. In this case, the WholeRowIterator approach wouldn't seem to be usable (because Accumulo only allows us to set filters for compaction time but not iterators).
>
> Here are our questions:
>
> (1) Has Accumulo's behavior when tearing down and re-initializing an iterator stack changed between 1.6.2 and the latest version? (I.e. is deepCopy now called?)

To the best of my knowledge, this has not been changed at all.

> (2) Are there any other ways in which storing state across iterator stack teardowns has been made any easier?

In short, no. If you can constrain your problem to only be in the 
context of a row and you can guarantee that you can read the row in 
memory, you're fine.

If you can't make this guarantee, you could *try* to use external 
systems to re-initialize some state during init(), but I would be 
worried about the cost of doing this efficiently. I think it would be 
very difficult to do correctly/efficiently/safely.

If you need to do cross-row operations, look into 
https://fluo.incubator.apache.org.

> (3) If not, are there any other tricks/hacks which we might consider using (albeit with caution) to store state or otherwise accomplish this? (Options we've mused about include figuring out another way for the iterators to store state beyond themselves -- can iterators write to the IteratorEnvironment to influence future iterator instantiations? -- and/or allowing the iterators to seek backwards to get the 'filtering key' they need.)

I don't have anything extra to add.

> (4) Also: any downsides to using the WholeRowIterator we should keep in mind?

Just the pressure of the holding the serialized row in memory and a 
non-zero cost of serializing/deserializing the row into the single 
key-value pair.

> Thanks in advance,
>
> Jonathan
>

The only other thing I'd suggest is investing the time to automate 
testing with "non-trivial" datasets. The biggest problem with Iterators 
is when Accumulo invokes them in ways that were unexpected by the 
developers. Much of the time, this comes as a part of the data scale 
(size of rows in relation to table.scan.max.memory). Putting your 
iterator through its paces now will greatly help when putting this into 
production.

- Josh

Re: using stateful iterators to do filtering during major compaction?

[Adam Fuchs <af...@apache.org>]

Hi Jonathan,

Did you consider simply not using the whole row iterator in the compaction
scope? Iterators don't get torn down and rebuilt in the compaction scope,
so partial row reads as you describe in the scan scope don't really exist
in compaction scopes unless another iterator above your filter is seeking.
(as an aside, the iterator environment will tell you if you're in a full or
partial compaction with respect to completeness of data, so plan
accordingly)

Other potential places to cache state:
1. The last key returned will be the first key seeked (non-inclusive) after
an iterator is torn down and reconstructed. You might be able to cache a
little bit of info at the end of the column qualifier.
2. Static soft value hash maps might help, using a combination of an
iterator option and the seeked range as the key.

Cheers,
Adam


On Mon, Apr 10, 2017 at 12:14 PM, Jonathan LASKO <
jonathan.lasko@raytheon.com> wrote:

> Hi Accumulo wizards,
>
> TL;DR - this is a question about custom iterators and saving state (or
> seeking backwards) in order to filter / mask data during major compaction.
>
> For a project I'm working on, we would like to be able to use one entry to
> filter other entries in the same row. (I will call the first entry the
> 'filtering key.') To do this, we would ensure that this 'filtering key'
> lexicographically precedes the other entries it would be used on.
>
> There is, of course, a "snag" with this idea: the iterator could simply
> read and save in memory the entry and then use it for subsequent filtering,
> were it not for the fact that the iterator stack can be dropped and
> re-initialized at any point in the row, including cf's/cq's that are
> already past the 'filtering key.' Our understanding is that the tserver
> processes can (and do!) restart and re-initialize the iterator stack at any
> point. When this happens, the tserver will "seek(...)" the newly
> re-initialized iterator stack back to the same row/cf/cq that the previous
> incarnation of the stack was on when it got re-initialized.
>
> When this teardown/re-init happens, the tserver doesn't call deepCopy(...)
> on the iterator stack; it just calls init(...). (At least, this is our
> experience in Accumulo 1.6.2.) For this reason, it is seen as a risky
> proposition to try to keep state in the iterators. (Josh Elser acknowledges
> this in his presentation on designing and testing custom iterators for
> Accumulo, https://www.slideshare.net/je2451/designing-and-testing-
> accumulo-iterators).
>
> Nevertheless, for the scantime scope, I believe we can use
> WholeRowIterator to ensure that we don't ever return data for a row until
> we've read the entire row, thus avoiding the need to keep state in the
> iterators. (If the iterator stack gets re-initialized, we should start over
> from the beginning of the row.)
>
> Our problem comes when we want to use this filter in majc.compaction scope
> to actually filter the masked data out of the system entirely. In this
> case, the WholeRowIterator approach wouldn't seem to be usable (because
> Accumulo only allows us to set filters for compaction time but not
> iterators).
>
> Here are our questions:
>
> (1) Has Accumulo's behavior when tearing down and re-initializing an
> iterator stack changed between 1.6.2 and the latest version? (I.e. is
> deepCopy now called?)
>
> (2) Are there any other ways in which storing state across iterator stack
> teardowns has been made any easier?
>
> (3) If not, are there any other tricks/hacks which we might consider using
> (albeit with caution) to store state or otherwise accomplish this? (Options
> we've mused about include figuring out another way for the iterators to
> store state beyond themselves -- can iterators write to the
> IteratorEnvironment to influence future iterator instantiations? -- and/or
> allowing the iterators to seek backwards to get the 'filtering key' they
> need.)
>
> (4) Also: any downsides to using the WholeRowIterator we should keep in
> mind?
>
> Thanks in advance,
>
> Jonathan
>