You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Russ Weeks <rw...@newbrightidea.com> on 2014/03/18 23:38:45 UTC

Deep-copying RowFilter

Hi,

org.apache.accumulo.core.iterators.user.RowFilter doesn't have a deepCopy
method, which seems to mean that I can't chain multiple RowFilters together.

Looking at some examples (GrepIterator, SortedKeyIterator) it seems pretty
easy to provide an implementation of deepCopy, the only thing that concerns
me is this part of the javadoc: "Creates a deep copy of this iterator *as
though seek had not yet been called*". There are a bunch of conditions
where RowFilter.skipRows will call seek() on the source iterator and
decision iterator.

Is the missing implementation of deepCopy just an oversight or are there
complications here? Where should I look to better understand how/when/why
accumulo will call my iterator's deepCopy method?

Thanks,
-Russ

Re: Deep-copying RowFilter

Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Mar 19, 2014 at 12:35 PM, Russ Weeks <rw...@newbrightidea.com>wrote:

> Hi, Keith,
>
> Thanks for your response.  I opened ACCUMULO-2501 and included a patch
> that works for me.
>
> The problem is that RowFilter calls deepCopy when it sets up its internal
> decisionIterator, so if the source iterator is also a RowFilter I'm hooped.
>

I commented on the ticket that this use case would make a nice unit test


>
> -Russ
>
>
> On Wed, Mar 19, 2014 at 9:24 AM, Keith Turner <ke...@deenlo.com> wrote:
>
>>
>>
>>
>> On Tue, Mar 18, 2014 at 6:38 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
>>
>>> Hi,
>>>
>>> org.apache.accumulo.core.iterators.user.RowFilter doesn't have a
>>> deepCopy method, which seems to mean that I can't chain multiple RowFilters
>>> together.
>>>
>>
>>> Looking at some examples (GrepIterator, SortedKeyIterator) it seems
>>> pretty easy to provide an implementation of deepCopy, the only thing that
>>> concerns me is this part of the javadoc: "Creates a deep copy of this
>>> iterator *as though seek had not yet been called*". There are a bunch of
>>> conditions where RowFilter.skipRows will call seek() on the source iterator
>>> and decision iterator.
>>>
>>
>> This is tricky. May need to open a ticket about making this easier.  I
>> think RowFilter should have a protected constructor to aid w/ deepCopy().
>> If you need to implement deepCopy (you may not see comments below), then I
>> think you would need to do something like the following.
>>
>> public class SomeRowFilter extends RowFilter {
>>
>>   private Map<String,String> options;
>>
>>   @Override
>>   public SortedKeyValueIterator<Key,Value> deepCopy(IteratorEnvironment
>> env) {
>>     SortedKeyValueIterator<Key,Value> dci = getSource().deepCopy(env);
>>     SomeRowFilter srf = new SomeRowFilter();
>>     srf.init(dci, options, env);
>>     return srf;
>>   }
>>
>>   @Override
>>   public boolean acceptRow(SortedKeyValueIterator<Key,Value> rowIterator)
>> throws IOException {
>>     // TODO Auto-generated method stub
>>     return false;
>>   }
>>
>>   @Override
>>   public void init(SortedKeyValueIterator<Key,Value> source,
>> Map<String,String> options, IteratorEnvironment env) throws IOException {
>>     super.init(source, options, env);
>>     this.options = options;
>>   }
>>
>> }
>>
>>
>>
>>
>>>
>>> Is the missing implementation of deepCopy just an oversight or are there
>>> complications here? Where should I look to better understand how/when/why
>>> accumulo will call my iterator's deepCopy method?
>>>
>>
>> Only a few iterators call deepCopy().  You can chain iterators that do
>> not implement deepCopy() w/ no problem as long as no iterator tries to call
>> it.   For example, the IntersectingIterator calls deepCopy() in its init
>> method (so that it can have multiple independent pointers into a row).  If
>> you wrapped a RowFilter with IntersectingIterator then your RowFilter would
>> need to implement deepCopy().
>>
>>
>>>
>>> Thanks,
>>> -Russ
>>>
>>
>>
>

Re: Deep-copying RowFilter

Posted by Russ Weeks <rw...@newbrightidea.com>.
Hi, Keith,

Thanks for your response.  I opened ACCUMULO-2501 and included a patch that
works for me.

The problem is that RowFilter calls deepCopy when it sets up its internal
decisionIterator, so if the source iterator is also a RowFilter I'm hooped.

-Russ


On Wed, Mar 19, 2014 at 9:24 AM, Keith Turner <ke...@deenlo.com> wrote:

>
>
>
> On Tue, Mar 18, 2014 at 6:38 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
>
>> Hi,
>>
>> org.apache.accumulo.core.iterators.user.RowFilter doesn't have a deepCopy
>> method, which seems to mean that I can't chain multiple RowFilters together.
>>
>
>> Looking at some examples (GrepIterator, SortedKeyIterator) it seems
>> pretty easy to provide an implementation of deepCopy, the only thing that
>> concerns me is this part of the javadoc: "Creates a deep copy of this
>> iterator *as though seek had not yet been called*". There are a bunch of
>> conditions where RowFilter.skipRows will call seek() on the source iterator
>> and decision iterator.
>>
>
> This is tricky. May need to open a ticket about making this easier.  I
> think RowFilter should have a protected constructor to aid w/ deepCopy().
> If you need to implement deepCopy (you may not see comments below), then I
> think you would need to do something like the following.
>
> public class SomeRowFilter extends RowFilter {
>
>   private Map<String,String> options;
>
>   @Override
>   public SortedKeyValueIterator<Key,Value> deepCopy(IteratorEnvironment
> env) {
>     SortedKeyValueIterator<Key,Value> dci = getSource().deepCopy(env);
>     SomeRowFilter srf = new SomeRowFilter();
>     srf.init(dci, options, env);
>     return srf;
>   }
>
>   @Override
>   public boolean acceptRow(SortedKeyValueIterator<Key,Value> rowIterator)
> throws IOException {
>     // TODO Auto-generated method stub
>     return false;
>   }
>
>   @Override
>   public void init(SortedKeyValueIterator<Key,Value> source,
> Map<String,String> options, IteratorEnvironment env) throws IOException {
>     super.init(source, options, env);
>     this.options = options;
>   }
>
> }
>
>
>
>
>>
>> Is the missing implementation of deepCopy just an oversight or are there
>> complications here? Where should I look to better understand how/when/why
>> accumulo will call my iterator's deepCopy method?
>>
>
> Only a few iterators call deepCopy().  You can chain iterators that do not
> implement deepCopy() w/ no problem as long as no iterator tries to call
> it.   For example, the IntersectingIterator calls deepCopy() in its init
> method (so that it can have multiple independent pointers into a row).  If
> you wrapped a RowFilter with IntersectingIterator then your RowFilter would
> need to implement deepCopy().
>
>
>>
>> Thanks,
>> -Russ
>>
>
>

Re: Deep-copying RowFilter

Posted by Keith Turner <ke...@deenlo.com>.
On Tue, Mar 18, 2014 at 6:38 PM, Russ Weeks <rw...@newbrightidea.com>wrote:

> Hi,
>
> org.apache.accumulo.core.iterators.user.RowFilter doesn't have a deepCopy
> method, which seems to mean that I can't chain multiple RowFilters together.
>

> Looking at some examples (GrepIterator, SortedKeyIterator) it seems pretty
> easy to provide an implementation of deepCopy, the only thing that concerns
> me is this part of the javadoc: "Creates a deep copy of this iterator *as
> though seek had not yet been called*". There are a bunch of conditions
> where RowFilter.skipRows will call seek() on the source iterator and
> decision iterator.
>

This is tricky. May need to open a ticket about making this easier.  I
think RowFilter should have a protected constructor to aid w/ deepCopy().
If you need to implement deepCopy (you may not see comments below), then I
think you would need to do something like the following.

public class SomeRowFilter extends RowFilter {

  private Map<String,String> options;

  @Override
  public SortedKeyValueIterator<Key,Value> deepCopy(IteratorEnvironment
env) {
    SortedKeyValueIterator<Key,Value> dci = getSource().deepCopy(env);
    SomeRowFilter srf = new SomeRowFilter();
    srf.init(dci, options, env);
    return srf;
  }

  @Override
  public boolean acceptRow(SortedKeyValueIterator<Key,Value> rowIterator)
throws IOException {
    // TODO Auto-generated method stub
    return false;
  }

  @Override
  public void init(SortedKeyValueIterator<Key,Value> source,
Map<String,String> options, IteratorEnvironment env) throws IOException {
    super.init(source, options, env);
    this.options = options;
  }

}




>
> Is the missing implementation of deepCopy just an oversight or are there
> complications here? Where should I look to better understand how/when/why
> accumulo will call my iterator's deepCopy method?
>

Only a few iterators call deepCopy().  You can chain iterators that do not
implement deepCopy() w/ no problem as long as no iterator tries to call
it.   For example, the IntersectingIterator calls deepCopy() in its init
method (so that it can have multiple independent pointers into a row).  If
you wrapped a RowFilter with IntersectingIterator then your RowFilter would
need to implement deepCopy().


>
> Thanks,
> -Russ
>