You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Trevor Adams <tr...@gmail.com> on 2011/12/09 22:50:56 UTC
Pig Tuples and Scanner
So I am looking to create a LoadFunc for Accumulo, and am just wondering
what would be the "correct" way to do this, here is my current plan.
Pig tuples are a set of columns for one given row in Accumulo, creating the
tuples with the Scanner seems possibly a bit odd. Loop over the elements
that it gives out (column value pairs) and fold/reduce on the rowid and
create some intermediate element that is used in a pseudo InputFormat of
<Row, ColVals> that can be used in the LoadFunc.
Since I don't understand some of the stuff in Accumulo, there may be a
better way to accomplish the above. If there is, great, otherwise I will
begin on the above.
-Trevor
Re: Pig Tuples and Scanner
Posted by Joey Daughtery <jd...@t-sciences.com>.
Trevor
Unfortunately, I have very little understanding of Accumulo myself.
Currently, I only understand the basics of it. And the
documentation/javadoc for Accumulo is not that good.
I would suggest posting to the Accumulo mailing list at
http://incubator.apache.org/accumulo/mailing_list.html
They seem to be very responsive. But I have come to realize that much of
what you can do with sql, does not translate to Accumulo without much
development.
Joe
On Fri, Dec 9, 2011 at 4:50 PM, Trevor Adams <tr...@gmail.com> wrote:
> So I am looking to create a LoadFunc for Accumulo, and am just wondering
> what would be the "correct" way to do this, here is my current plan.
>
> Pig tuples are a set of columns for one given row in Accumulo, creating
> the tuples with the Scanner seems possibly a bit odd. Loop over the
> elements that it gives out (column value pairs) and fold/reduce on the
> rowid and create some intermediate element that is used in a pseudo
> InputFormat of <Row, ColVals> that can be used in the LoadFunc.
>
> Since I don't understand some of the stuff in Accumulo, there may be a
> better way to accomplish the above. If there is, great, otherwise I will
> begin on the above.
>
> -Trevor
>
Re: Pig Tuples and Scanner
Posted by Adam Fuchs <ad...@ugov.gov>.
Ah, I see. I missed that you were starting from HBase. The long rows would
be a problem for HBase too, although I believe HBase is pretty much
restricted to rows that fit into memory due to the way that results are
batched. Please keep us apprised of your progress.
Cheers,
Adam
On Mon, Dec 12, 2011 at 12:09 PM, Trevor Adams <tr...@gmail.com>wrote:
> Adam,
>
> I can see how the large number of columns can be a problem but I also
> think this would be an issue for HBase as well. While I understand it is a
> different project this looks like it could be a problem for them as well.
> Prior to the most recent version of pig (I believe this is when it was
> added) you had to specify exact column names (cf:qual pairs), only recently
> did they add support for grabbing an entire column family from a row. I
> will explore my options a bit, and see what happens I guess. I will
> probably start with the specific loader and then try to generalize from
> there. Thanks
>
> -Trevor
>
> On Mon, Dec 12, 2011 at 9:10 AM, Adam Fuchs <ad...@ugov.gov> wrote:
>
>> Trevor,
>>
>> I think there are a few different ways you could implement a LoadFunc on
>> top of Accumulo. The most basic and universal option might be to use a
>> single entry (Key/Value pair) as a Pig tuple. This is easier to code, but
>> it might not correspond to your objects if you split your objects out into
>> multiple columns, with one row per object. You might be able to use a Pig
>> operator to group your data after using this type of LoadFunc, or you might
>> want to create a more customized LoadFunc that understands more about how
>> you organize your data in Accumulo.
>>
>> The second option that I've seen is the one that you describe -- namely
>> iterating over the columns in each row to produce a row tuple. The way to
>> do this is really just to loop over the elements returned by a Scanner and
>> pinch off tuples when you see a new row ID. If you use the same splitting
>> strategy that AccumuloInputFormat uses (i.e. pick split points from
>> tablets) then rows will never be split across multiple splits. A single
>> Scanner per RecordReader should work great for you, and I've seen other
>> people implement LoadFunc successfully in that way.
>>
>> One thing to watch out for is that a single row in Accumulo could
>> potentially have hundres of millions or even billions of columns. Using a
>> row-based tuple for you LoadFunc could result in your application running
>> out of memory if you try to process any arbitrary table. We commonly see
>> this when looking at graph structures where edges are represented by
>> columns. Zipfian distributions can make for some very big rows. This just
>> means you have to be a bit careful about what you try to pull into a tuple.
>>
>> Cheers,
>> Adam
>>
>>
>> On Fri, Dec 9, 2011 at 4:50 PM, Trevor Adams <tr...@gmail.com>wrote:
>>
>>> So I am looking to create a LoadFunc for Accumulo, and am just wondering
>>> what would be the "correct" way to do this, here is my current plan.
>>>
>>> Pig tuples are a set of columns for one given row in Accumulo, creating
>>> the tuples with the Scanner seems possibly a bit odd. Loop over the
>>> elements that it gives out (column value pairs) and fold/reduce on the
>>> rowid and create some intermediate element that is used in a pseudo
>>> InputFormat of <Row, ColVals> that can be used in the LoadFunc.
>>>
>>> Since I don't understand some of the stuff in Accumulo, there may be a
>>> better way to accomplish the above. If there is, great, otherwise I will
>>> begin on the above.
>>>
>>> -Trevor
>>>
>>
>>
>
Re: Pig Tuples and Scanner
Posted by Trevor Adams <tr...@gmail.com>.
Adam,
I can see how the large number of columns can be a problem but I also think
this would be an issue for HBase as well. While I understand it is a
different project this looks like it could be a problem for them as well.
Prior to the most recent version of pig (I believe this is when it was
added) you had to specify exact column names (cf:qual pairs), only recently
did they add support for grabbing an entire column family from a row. I
will explore my options a bit, and see what happens I guess. I will
probably start with the specific loader and then try to generalize from
there. Thanks
-Trevor
On Mon, Dec 12, 2011 at 9:10 AM, Adam Fuchs <ad...@ugov.gov> wrote:
> Trevor,
>
> I think there are a few different ways you could implement a LoadFunc on
> top of Accumulo. The most basic and universal option might be to use a
> single entry (Key/Value pair) as a Pig tuple. This is easier to code, but
> it might not correspond to your objects if you split your objects out into
> multiple columns, with one row per object. You might be able to use a Pig
> operator to group your data after using this type of LoadFunc, or you might
> want to create a more customized LoadFunc that understands more about how
> you organize your data in Accumulo.
>
> The second option that I've seen is the one that you describe -- namely
> iterating over the columns in each row to produce a row tuple. The way to
> do this is really just to loop over the elements returned by a Scanner and
> pinch off tuples when you see a new row ID. If you use the same splitting
> strategy that AccumuloInputFormat uses (i.e. pick split points from
> tablets) then rows will never be split across multiple splits. A single
> Scanner per RecordReader should work great for you, and I've seen other
> people implement LoadFunc successfully in that way.
>
> One thing to watch out for is that a single row in Accumulo could
> potentially have hundres of millions or even billions of columns. Using a
> row-based tuple for you LoadFunc could result in your application running
> out of memory if you try to process any arbitrary table. We commonly see
> this when looking at graph structures where edges are represented by
> columns. Zipfian distributions can make for some very big rows. This just
> means you have to be a bit careful about what you try to pull into a tuple.
>
> Cheers,
> Adam
>
>
> On Fri, Dec 9, 2011 at 4:50 PM, Trevor Adams <tr...@gmail.com>wrote:
>
>> So I am looking to create a LoadFunc for Accumulo, and am just wondering
>> what would be the "correct" way to do this, here is my current plan.
>>
>> Pig tuples are a set of columns for one given row in Accumulo, creating
>> the tuples with the Scanner seems possibly a bit odd. Loop over the
>> elements that it gives out (column value pairs) and fold/reduce on the
>> rowid and create some intermediate element that is used in a pseudo
>> InputFormat of <Row, ColVals> that can be used in the LoadFunc.
>>
>> Since I don't understand some of the stuff in Accumulo, there may be a
>> better way to accomplish the above. If there is, great, otherwise I will
>> begin on the above.
>>
>> -Trevor
>>
>
>
Re: Filter use
Posted by Billie J Rinaldi <bi...@ugov.gov>.
On Friday, December 16, 2011 10:18:04 AM, "Joey Daughtery" <jd...@t-sciences.com> wrote:
> When I implement the accept method, is there an object inside the
> Filter object that I am extending that I should access to obtain the
> value that I am intending to base the implementation of accept on?
You will have to implement the init method and pull the information you need out of Map<String,String> options. See org.apache.accumulo.core.iterators.user.AgeOffFilter for an example.
Billie
Re: Filter use
Posted by Joey Daughtery <jd...@t-sciences.com>.
When I implement the accept method, is there an object inside the Filter
object that I am extending that I should access to obtain the value that I
am intending to base the implementation of accept on?
Thanks
Joe
On Fri, Dec 16, 2011 at 9:47 AM, Billie J Rinaldi <billie.j.rinaldi@ugov.gov
> wrote:
> You should place it in the accumulo/lib/ext directory. The classloader
> will pick up changes you make to jars in that directory.
>
> Billie
>
>
> ----- Original Message -----
> > From: "Joey Daughtery" <jd...@t-sciences.com>
> > To: accumulo-user@incubator.apache.org
> > Sent: Friday, December 16, 2011 9:26:19 AM
> > Subject: Re: Filter use
> > I think I have found the issue. It is throwing a class not found
> > exception on my subclass. I did not add it to the accumulo build. I
> > will do that and try again.
> >
> > I am guessing that I should be able to create a client jar with my
> > subclass and place it into the accumulo/lib directory.
> >
> >
> > Thanks
> >
> > Joe
> >
> >
> > On Fri, Dec 16, 2011 at 8:52 AM, Eric Newton < eric.newton@gmail.com >
> > wrote:
> >
> >
> > Do you see a stack trace in the tablet server log?
> >
> >
> >
> >
> >
> > On Fri, Dec 16, 2011 at 8:47 AM, Joey Daughtery <
> > jdaughtery@t-sciences.com > wrote:
> >
> >
> > All
> > I was finally able to get back to this. When I implemented the
> > following on Accumulo 1.4 and execute, I get the exception below.
> >
> > Also, where are the option keys defined for IteratorSetting.addOption?
> >
> > 1. I subclassed the org.apache.accumulo.core.iterators.Filter object
> > and implemented the accept(Key key, Value value) method.
> >
> > 2. Then I utilized it in the following way:
> > IteratorSetting is = new IteratorSetting(15, "idFilter",
> > fi.getClass());
> > is.addOption("valuesToSkip", userId);
> >
> >
> > Range r = new Range();
> > Scanner s = conn.createScanner("index_table", scanauths);
> > s.setRange(r);
> > s.addScanIterator(is);
> >
> >
> >
> > Caused by: org.apache.thrift.TApplicationException: Internal error
> > processing startScan
> > at
> >
> org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
> > at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
> > at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:67)
> > at $Proxy1.startScan(Unknown Source)
> > at
> >
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
> > at
> >
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
> > ... 4 more
> >
> > Thanks
> >
> > Joe
> >
> >
> >
> >
> > On Mon, Dec 12, 2011 at 10:23 AM, Billie J Rinaldi <
> > billie.j.rinaldi@ugov.gov > wrote:
> >
> >
> >
> > On Date: Dec 12, 2011 9:29 AM, "Aaron Cordova" <
> > aaron.cordova@interllective.com > wrote:
> > > 3. Configure your table to use your filtering iterator as described
> > > in
> > > this section of the user manual:
> > >
> http://incubator.apache.org/accumulo/user_manual_1.3-incubating/Table_Configuration.html
> > > , where you specify the name of your Java class as the first option
> > > in
> > > the Shell, rather than 'ageoff' or 'regex'.
> >
> > Unfortunately the manual does not say how to specify filters
> > programmatically. This is somewhat confusing in 1.3 because the Filter
> > class you will be writing is a plugin to the FilteringIterator, and
> > the Filter is not itself an Iterator. Configuring the filter in the
> > shell and just setting its parameters programmatically is a
> > possibility, but the syntax Aaron gave for setting the parameter is
> > incorrect. Here is an example of configuring a FilteringIterator:
> >
> > scanner.setScanIterators(priority,
> > "org.apache.accumulo.core.iterators.FilteringIterator", "myFilter");
> > scanner.setScanIteratorOption("myFilter", "0",
> > "org.apache.accumulo.core.iterators.filter.MyFilter");
> > scanner.setScanIteratorOption("myFilter", "0.valuesToSkip", "10");
> > scanner.setScanIteratorOption("myFilter", "1",
> > "org.apache.accumulo.core.iterators.filter.AgeOffFilter");
> > scanner.setScanIteratorOption("myFilter", "1.ttl", ttl);
> >
> > If you can start coding to 1.4 instead of 1.3, I would highly
> > recommend it. All provided user iterators in 1.4 are located in the
> > org.apache.accumulo.core.iterators.user package. Filters are Iterators
> > and are much easier to configure:
> >
> > IteratorSetting setting = new IteratorSetting(priority,
> > MyFilter.class);
> > setting.addOption("valuesToSkip", "10");
> > scanner.addScanIterator(setting);
> >
> > ** it is recommended to configure the IteratorSetting with static
> > methods instead of addOption,
> > e.g. TimestampFilter.setRange(setting, "20120101000000GMT",
> > "20120102000000GMT");
> >
> > One of the above options should work. However, filtering based on the
> > Value, the unsorted portion of a sorted key/value store, is not going
> > to be an efficient operation. Your query will have to scan over all
> > the entries in the table to find the Values you want. This might be ok
> > if the query returns most of the data in the table, but if it just
> > returns a small fraction of the data it would be better to figure out
> > how to get the sorting to work to your advantage.
> >
> > Billie
>
Re: Filter use
Posted by Billie J Rinaldi <bi...@ugov.gov>.
You should place it in the accumulo/lib/ext directory. The classloader will pick up changes you make to jars in that directory.
Billie
----- Original Message -----
> From: "Joey Daughtery" <jd...@t-sciences.com>
> To: accumulo-user@incubator.apache.org
> Sent: Friday, December 16, 2011 9:26:19 AM
> Subject: Re: Filter use
> I think I have found the issue. It is throwing a class not found
> exception on my subclass. I did not add it to the accumulo build. I
> will do that and try again.
>
> I am guessing that I should be able to create a client jar with my
> subclass and place it into the accumulo/lib directory.
>
>
> Thanks
>
> Joe
>
>
> On Fri, Dec 16, 2011 at 8:52 AM, Eric Newton < eric.newton@gmail.com >
> wrote:
>
>
> Do you see a stack trace in the tablet server log?
>
>
>
>
>
> On Fri, Dec 16, 2011 at 8:47 AM, Joey Daughtery <
> jdaughtery@t-sciences.com > wrote:
>
>
> All
> I was finally able to get back to this. When I implemented the
> following on Accumulo 1.4 and execute, I get the exception below.
>
> Also, where are the option keys defined for IteratorSetting.addOption?
>
> 1. I subclassed the org.apache.accumulo.core.iterators.Filter object
> and implemented the accept(Key key, Value value) method.
>
> 2. Then I utilized it in the following way:
> IteratorSetting is = new IteratorSetting(15, "idFilter",
> fi.getClass());
> is.addOption("valuesToSkip", userId);
>
>
> Range r = new Range();
> Scanner s = conn.createScanner("index_table", scanauths);
> s.setRange(r);
> s.addScanIterator(is);
>
>
>
> Caused by: org.apache.thrift.TApplicationException: Internal error
> processing startScan
> at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:67)
> at $Proxy1.startScan(Unknown Source)
> at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
> at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
> ... 4 more
>
> Thanks
>
> Joe
>
>
>
>
> On Mon, Dec 12, 2011 at 10:23 AM, Billie J Rinaldi <
> billie.j.rinaldi@ugov.gov > wrote:
>
>
>
> On Date: Dec 12, 2011 9:29 AM, "Aaron Cordova" <
> aaron.cordova@interllective.com > wrote:
> > 3. Configure your table to use your filtering iterator as described
> > in
> > this section of the user manual:
> > http://incubator.apache.org/accumulo/user_manual_1.3-incubating/Table_Configuration.html
> > , where you specify the name of your Java class as the first option
> > in
> > the Shell, rather than 'ageoff' or 'regex'.
>
> Unfortunately the manual does not say how to specify filters
> programmatically. This is somewhat confusing in 1.3 because the Filter
> class you will be writing is a plugin to the FilteringIterator, and
> the Filter is not itself an Iterator. Configuring the filter in the
> shell and just setting its parameters programmatically is a
> possibility, but the syntax Aaron gave for setting the parameter is
> incorrect. Here is an example of configuring a FilteringIterator:
>
> scanner.setScanIterators(priority,
> "org.apache.accumulo.core.iterators.FilteringIterator", "myFilter");
> scanner.setScanIteratorOption("myFilter", "0",
> "org.apache.accumulo.core.iterators.filter.MyFilter");
> scanner.setScanIteratorOption("myFilter", "0.valuesToSkip", "10");
> scanner.setScanIteratorOption("myFilter", "1",
> "org.apache.accumulo.core.iterators.filter.AgeOffFilter");
> scanner.setScanIteratorOption("myFilter", "1.ttl", ttl);
>
> If you can start coding to 1.4 instead of 1.3, I would highly
> recommend it. All provided user iterators in 1.4 are located in the
> org.apache.accumulo.core.iterators.user package. Filters are Iterators
> and are much easier to configure:
>
> IteratorSetting setting = new IteratorSetting(priority,
> MyFilter.class);
> setting.addOption("valuesToSkip", "10");
> scanner.addScanIterator(setting);
>
> ** it is recommended to configure the IteratorSetting with static
> methods instead of addOption,
> e.g. TimestampFilter.setRange(setting, "20120101000000GMT",
> "20120102000000GMT");
>
> One of the above options should work. However, filtering based on the
> Value, the unsorted portion of a sorted key/value store, is not going
> to be an efficient operation. Your query will have to scan over all
> the entries in the table to find the Values you want. This might be ok
> if the query returns most of the data in the table, but if it just
> returns a small fraction of the data it would be better to figure out
> how to get the sorting to work to your advantage.
>
> Billie
Re: Filter use
Posted by Joey Daughtery <jd...@t-sciences.com>.
I think I have found the issue. It is throwing a class not found exception
on my subclass. I did not add it to the accumulo build. I will do that
and try again.
I am guessing that I should be able to create a client jar with my subclass
and place it into the accumulo/lib directory.
Thanks
Joe
On Fri, Dec 16, 2011 at 8:52 AM, Eric Newton <er...@gmail.com> wrote:
> Do you see a stack trace in the tablet server log?
>
>
> On Fri, Dec 16, 2011 at 8:47 AM, Joey Daughtery <jdaughtery@t-sciences.com
> > wrote:
>
>> All
>> I was finally able to get back to this. When I implemented the following
>> on Accumulo 1.4 and execute, I get the exception below.
>>
>> Also, where are the option keys defined for IteratorSetting.addOption?
>>
>> 1. I subclassed the org.apache.accumulo.core.iterators.Filter object and
>> implemented the accept(Key key, Value value) method.
>>
>> 2. Then I utilized it in the following way:
>> IteratorSetting is = new IteratorSetting(15, "idFilter", fi.getClass());
>> is.addOption("valuesToSkip", userId);
>>
>>
>> Range r = new Range();
>> Scanner s = conn.createScanner("index_table", scanauths);
>> s.setRange(r);
>> s.addScanIterator(is);
>>
>>
>>
>> Caused by: org.apache.thrift.TApplicationException: Internal error
>> processing startScan
>> at
>> org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>> at
>> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>> at
>> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:67)
>> at $Proxy1.startScan(Unknown Source)
>> at
>> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>> at
>> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>> ... 4 more
>>
>> Thanks
>>
>> Joe
>>
>>
>> On Mon, Dec 12, 2011 at 10:23 AM, Billie J Rinaldi <
>> billie.j.rinaldi@ugov.gov> wrote:
>>
>>> On Date: Dec 12, 2011 9:29 AM, "Aaron Cordova" <
>>> aaron.cordova@interllective.com > wrote:
>>> > 3. Configure your table to use your filtering iterator as described in
>>> > this section of the user manual:
>>> >
>>> http://incubator.apache.org/accumulo/user_manual_1.3-incubating/Table_Configuration.html
>>> > , where you specify the name of your Java class as the first option in
>>> > the Shell, rather than 'ageoff' or 'regex'.
>>>
>>> Unfortunately the manual does not say how to specify filters
>>> programmatically. This is somewhat confusing in 1.3 because the Filter
>>> class you will be writing is a plugin to the FilteringIterator, and the
>>> Filter is not itself an Iterator. Configuring the filter in the shell and
>>> just setting its parameters programmatically is a possibility, but the
>>> syntax Aaron gave for setting the parameter is incorrect. Here is an
>>> example of configuring a FilteringIterator:
>>>
>>> scanner.setScanIterators(priority,
>>> "org.apache.accumulo.core.iterators.FilteringIterator", "myFilter");
>>> scanner.setScanIteratorOption("myFilter", "0",
>>> "org.apache.accumulo.core.iterators.filter.MyFilter");
>>> scanner.setScanIteratorOption("myFilter", "0.valuesToSkip", "10");
>>> scanner.setScanIteratorOption("myFilter", "1",
>>> "org.apache.accumulo.core.iterators.filter.AgeOffFilter");
>>> scanner.setScanIteratorOption("myFilter", "1.ttl", ttl);
>>>
>>> If you can start coding to 1.4 instead of 1.3, I would highly recommend
>>> it. All provided user iterators in 1.4 are located in the
>>> org.apache.accumulo.core.iterators.user package. Filters are Iterators and
>>> are much easier to configure:
>>>
>>> IteratorSetting setting = new IteratorSetting(priority, MyFilter.class);
>>> setting.addOption("valuesToSkip", "10");
>>> scanner.addScanIterator(setting);
>>>
>>> ** it is recommended to configure the IteratorSetting with static
>>> methods instead of addOption,
>>> e.g. TimestampFilter.setRange(setting, "20120101000000GMT",
>>> "20120102000000GMT");
>>>
>>> One of the above options should work. However, filtering based on the
>>> Value, the unsorted portion of a sorted key/value store, is not going to be
>>> an efficient operation. Your query will have to scan over all the entries
>>> in the table to find the Values you want. This might be ok if the query
>>> returns most of the data in the table, but if it just returns a small
>>> fraction of the data it would be better to figure out how to get the
>>> sorting to work to your advantage.
>>>
>>> Billie
>>>
>>
>>
>
Re: Filter use
Posted by Eric Newton <er...@gmail.com>.
Do you see a stack trace in the tablet server log?
On Fri, Dec 16, 2011 at 8:47 AM, Joey Daughtery
<jd...@t-sciences.com>wrote:
> All
> I was finally able to get back to this. When I implemented the following
> on Accumulo 1.4 and execute, I get the exception below.
>
> Also, where are the option keys defined for IteratorSetting.addOption?
>
> 1. I subclassed the org.apache.accumulo.core.iterators.Filter object and
> implemented the accept(Key key, Value value) method.
>
> 2. Then I utilized it in the following way:
> IteratorSetting is = new IteratorSetting(15, "idFilter", fi.getClass());
> is.addOption("valuesToSkip", userId);
>
>
> Range r = new Range();
> Scanner s = conn.createScanner("index_table", scanauths);
> s.setRange(r);
> s.addScanIterator(is);
>
>
>
> Caused by: org.apache.thrift.TApplicationException: Internal error
> processing startScan
> at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:67)
> at $Proxy1.startScan(Unknown Source)
> at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
> at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
> ... 4 more
>
> Thanks
>
> Joe
>
>
> On Mon, Dec 12, 2011 at 10:23 AM, Billie J Rinaldi <
> billie.j.rinaldi@ugov.gov> wrote:
>
>> On Date: Dec 12, 2011 9:29 AM, "Aaron Cordova" <
>> aaron.cordova@interllective.com > wrote:
>> > 3. Configure your table to use your filtering iterator as described in
>> > this section of the user manual:
>> >
>> http://incubator.apache.org/accumulo/user_manual_1.3-incubating/Table_Configuration.html
>> > , where you specify the name of your Java class as the first option in
>> > the Shell, rather than 'ageoff' or 'regex'.
>>
>> Unfortunately the manual does not say how to specify filters
>> programmatically. This is somewhat confusing in 1.3 because the Filter
>> class you will be writing is a plugin to the FilteringIterator, and the
>> Filter is not itself an Iterator. Configuring the filter in the shell and
>> just setting its parameters programmatically is a possibility, but the
>> syntax Aaron gave for setting the parameter is incorrect. Here is an
>> example of configuring a FilteringIterator:
>>
>> scanner.setScanIterators(priority,
>> "org.apache.accumulo.core.iterators.FilteringIterator", "myFilter");
>> scanner.setScanIteratorOption("myFilter", "0",
>> "org.apache.accumulo.core.iterators.filter.MyFilter");
>> scanner.setScanIteratorOption("myFilter", "0.valuesToSkip", "10");
>> scanner.setScanIteratorOption("myFilter", "1",
>> "org.apache.accumulo.core.iterators.filter.AgeOffFilter");
>> scanner.setScanIteratorOption("myFilter", "1.ttl", ttl);
>>
>> If you can start coding to 1.4 instead of 1.3, I would highly recommend
>> it. All provided user iterators in 1.4 are located in the
>> org.apache.accumulo.core.iterators.user package. Filters are Iterators and
>> are much easier to configure:
>>
>> IteratorSetting setting = new IteratorSetting(priority, MyFilter.class);
>> setting.addOption("valuesToSkip", "10");
>> scanner.addScanIterator(setting);
>>
>> ** it is recommended to configure the IteratorSetting with static methods
>> instead of addOption,
>> e.g. TimestampFilter.setRange(setting, "20120101000000GMT",
>> "20120102000000GMT");
>>
>> One of the above options should work. However, filtering based on the
>> Value, the unsorted portion of a sorted key/value store, is not going to be
>> an efficient operation. Your query will have to scan over all the entries
>> in the table to find the Values you want. This might be ok if the query
>> returns most of the data in the table, but if it just returns a small
>> fraction of the data it would be better to figure out how to get the
>> sorting to work to your advantage.
>>
>> Billie
>>
>
>
Re: Filter use
Posted by Joey Daughtery <jd...@t-sciences.com>.
All
I was finally able to get back to this. When I implemented the following
on Accumulo 1.4 and execute, I get the exception below.
Also, where are the option keys defined for IteratorSetting.addOption?
1. I subclassed the org.apache.accumulo.core.iterators.Filter object and
implemented the accept(Key key, Value value) method.
2. Then I utilized it in the following way:
IteratorSetting is = new IteratorSetting(15, "idFilter", fi.getClass());
is.addOption("valuesToSkip", userId);
Range r = new Range();
Scanner s = conn.createScanner("index_table", scanauths);
s.setRange(r);
s.addScanIterator(is);
Caused by: org.apache.thrift.TApplicationException: Internal error
processing startScan
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:67)
at $Proxy1.startScan(Unknown Source)
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
... 4 more
Thanks
Joe
On Mon, Dec 12, 2011 at 10:23 AM, Billie J Rinaldi <
billie.j.rinaldi@ugov.gov> wrote:
> On Date: Dec 12, 2011 9:29 AM, "Aaron Cordova" <
> aaron.cordova@interllective.com > wrote:
> > 3. Configure your table to use your filtering iterator as described in
> > this section of the user manual:
> >
> http://incubator.apache.org/accumulo/user_manual_1.3-incubating/Table_Configuration.html
> > , where you specify the name of your Java class as the first option in
> > the Shell, rather than 'ageoff' or 'regex'.
>
> Unfortunately the manual does not say how to specify filters
> programmatically. This is somewhat confusing in 1.3 because the Filter
> class you will be writing is a plugin to the FilteringIterator, and the
> Filter is not itself an Iterator. Configuring the filter in the shell and
> just setting its parameters programmatically is a possibility, but the
> syntax Aaron gave for setting the parameter is incorrect. Here is an
> example of configuring a FilteringIterator:
>
> scanner.setScanIterators(priority,
> "org.apache.accumulo.core.iterators.FilteringIterator", "myFilter");
> scanner.setScanIteratorOption("myFilter", "0",
> "org.apache.accumulo.core.iterators.filter.MyFilter");
> scanner.setScanIteratorOption("myFilter", "0.valuesToSkip", "10");
> scanner.setScanIteratorOption("myFilter", "1",
> "org.apache.accumulo.core.iterators.filter.AgeOffFilter");
> scanner.setScanIteratorOption("myFilter", "1.ttl", ttl);
>
> If you can start coding to 1.4 instead of 1.3, I would highly recommend
> it. All provided user iterators in 1.4 are located in the
> org.apache.accumulo.core.iterators.user package. Filters are Iterators and
> are much easier to configure:
>
> IteratorSetting setting = new IteratorSetting(priority, MyFilter.class);
> setting.addOption("valuesToSkip", "10");
> scanner.addScanIterator(setting);
>
> ** it is recommended to configure the IteratorSetting with static methods
> instead of addOption,
> e.g. TimestampFilter.setRange(setting, "20120101000000GMT",
> "20120102000000GMT");
>
> One of the above options should work. However, filtering based on the
> Value, the unsorted portion of a sorted key/value store, is not going to be
> an efficient operation. Your query will have to scan over all the entries
> in the table to find the Values you want. This might be ok if the query
> returns most of the data in the table, but if it just returns a small
> fraction of the data it would be better to figure out how to get the
> sorting to work to your advantage.
>
> Billie
>
Re: Filter use
Posted by Billie J Rinaldi <bi...@ugov.gov>.
On Date: Dec 12, 2011 9:29 AM, "Aaron Cordova" < aaron.cordova@interllective.com > wrote:
> 3. Configure your table to use your filtering iterator as described in
> this section of the user manual:
> http://incubator.apache.org/accumulo/user_manual_1.3-incubating/Table_Configuration.html
> , where you specify the name of your Java class as the first option in
> the Shell, rather than 'ageoff' or 'regex'.
Unfortunately the manual does not say how to specify filters programmatically. This is somewhat confusing in 1.3 because the Filter class you will be writing is a plugin to the FilteringIterator, and the Filter is not itself an Iterator. Configuring the filter in the shell and just setting its parameters programmatically is a possibility, but the syntax Aaron gave for setting the parameter is incorrect. Here is an example of configuring a FilteringIterator:
scanner.setScanIterators(priority, "org.apache.accumulo.core.iterators.FilteringIterator", "myFilter");
scanner.setScanIteratorOption("myFilter", "0", "org.apache.accumulo.core.iterators.filter.MyFilter");
scanner.setScanIteratorOption("myFilter", "0.valuesToSkip", "10");
scanner.setScanIteratorOption("myFilter", "1", "org.apache.accumulo.core.iterators.filter.AgeOffFilter");
scanner.setScanIteratorOption("myFilter", "1.ttl", ttl);
If you can start coding to 1.4 instead of 1.3, I would highly recommend it. All provided user iterators in 1.4 are located in the org.apache.accumulo.core.iterators.user package. Filters are Iterators and are much easier to configure:
IteratorSetting setting = new IteratorSetting(priority, MyFilter.class);
setting.addOption("valuesToSkip", "10");
scanner.addScanIterator(setting);
** it is recommended to configure the IteratorSetting with static methods instead of addOption,
e.g. TimestampFilter.setRange(setting, "20120101000000GMT", "20120102000000GMT");
One of the above options should work. However, filtering based on the Value, the unsorted portion of a sorted key/value store, is not going to be an efficient operation. Your query will have to scan over all the entries in the table to find the Values you want. This might be ok if the query returns most of the data in the table, but if it just returns a small fraction of the data it would be better to figure out how to get the sorting to work to your advantage.
Billie
Re: Filter use
Posted by Adam Fuchs <ad...@ugov.gov>.
Message from Aaron below.
Adam
---------- Forwarded message ----------
From: "Aaron Cordova" <aa...@interllective.com>
Date: Dec 12, 2011 9:29 AM
Subject: Re: Filter use
To: "Adam Fuchs" <ad...@ugov.gov>
Joey,
It is possible to write your own filtering iterators to do what you
described. The idea would be to pass in some options to the iterator at
scan-time, such as the values that you want to filter out, e.g. '10' in
your example, and the filtering iterator would apply your logic as you
perform a full table scan.
To write your own filtering iterator use the following as a guide:
1. create a Java class that implements
org.apache.accumulo.core.iterators.filter.Filter and that contains your
filtering logic. The 'init' function lets you save options set by the
client scanner. The 'accept' function is where in your example, you'd just
be comparing all values to the values passed in as options from the scanner
and skipping those that match (i.e. return 'false').
2. Create a jar containing your class and add it to the $ACCUMULO_HOME/lib
directory on every machine in your cluster
3. Configure your table to use your filtering iterator as described in this
section of the user manual:
http://incubator.apache.org/accumulo/user_manual_1.3-incubating/Table_Configuration.html,
where you specify the name of your Java class as the first option in the
Shell, rather than 'ageoff' or 'regex'.
4. Write your Accumulo Java clients to specify values you want to skip as
options to the scanner - i.e:
scanner.setScanIteratorOption("myFilter", "valuesToSkip", "10")
But be sure to do any necessary transformations on your values, since you
can only pass strings as options to your filter. You'd have to parse the
string into an Integer to match any serialized integers you may have stored
as values in your table.
Aaron
Re: Pig Tuples and Scanner
Posted by Adam Fuchs <ad...@ugov.gov>.
Trevor,
I think there are a few different ways you could implement a LoadFunc on
top of Accumulo. The most basic and universal option might be to use a
single entry (Key/Value pair) as a Pig tuple. This is easier to code, but
it might not correspond to your objects if you split your objects out into
multiple columns, with one row per object. You might be able to use a Pig
operator to group your data after using this type of LoadFunc, or you might
want to create a more customized LoadFunc that understands more about how
you organize your data in Accumulo.
The second option that I've seen is the one that you describe -- namely
iterating over the columns in each row to produce a row tuple. The way to
do this is really just to loop over the elements returned by a Scanner and
pinch off tuples when you see a new row ID. If you use the same splitting
strategy that AccumuloInputFormat uses (i.e. pick split points from
tablets) then rows will never be split across multiple splits. A single
Scanner per RecordReader should work great for you, and I've seen other
people implement LoadFunc successfully in that way.
One thing to watch out for is that a single row in Accumulo could
potentially have hundres of millions or even billions of columns. Using a
row-based tuple for you LoadFunc could result in your application running
out of memory if you try to process any arbitrary table. We commonly see
this when looking at graph structures where edges are represented by
columns. Zipfian distributions can make for some very big rows. This just
means you have to be a bit careful about what you try to pull into a tuple.
Cheers,
Adam
On Fri, Dec 9, 2011 at 4:50 PM, Trevor Adams <tr...@gmail.com> wrote:
> So I am looking to create a LoadFunc for Accumulo, and am just wondering
> what would be the "correct" way to do this, here is my current plan.
>
> Pig tuples are a set of columns for one given row in Accumulo, creating
> the tuples with the Scanner seems possibly a bit odd. Loop over the
> elements that it gives out (column value pairs) and fold/reduce on the
> rowid and create some intermediate element that is used in a pseudo
> InputFormat of <Row, ColVals> that can be used in the LoadFunc.
>
> Since I don't understand some of the stuff in Accumulo, there may be a
> better way to accomplish the above. If there is, great, otherwise I will
> begin on the above.
>
> -Trevor
>