You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Aaron Cordova <aa...@cordovas.org> on 2012/01/02 15:12:05 UTC
upgrade from 1.3 to 1.4
Is there anything that needs to be done to upgrade from 1.3 to 1.4?
Re: upgrade from 1.3 to 1.4
Posted by Billie J Rinaldi <bi...@ugov.gov>.
On Tuesday, January 3, 2012 12:51:20 PM, "Aaron Cordova" <aa...@cordovas.org> wrote:
> yeah in this case i suppose there's not much that can go wrong. even
> if both combiners are turned off at some point, when the
> summingcombiner is turned on it should do the right thing. As long as
> the values that get written don't get transformed differently, i.e. as
> long as the functionality is really identical.
I wrote a test to check the new encodings against the old ones, but if you see anything wrong do let us know.
Billie
Re: upgrade from 1.3 to 1.4
Posted by Aaron Cordova <aa...@cordovas.org>.
yeah in this case i suppose there's not much that can go wrong. even if both combiners are turned off at some point, when the summingcombiner is turned on it should do the right thing. As long as the values that get written don't get transformed differently, i.e. as long as the functionality is really identical.
On Jan 3, 2012, at 12:45 PM, Adam Fuchs wrote:
> Oh yeah, and they're (mutually) idempotent too, right?
>
> Adam
>
>
> On Tue, Jan 3, 2012 at 12:43 PM, Adam Fuchs <ad...@ugov.gov> wrote:
> Why not add the combiner (so that both are enabled) and then remove the aggregator? The two iterators should be commutative and associative, so results should be reasonable and compactions should be safe when both are enabled in the tree.
>
> Adam
>
>
> On Tue, Jan 3, 2012 at 12:39 PM, Billie J Rinaldi <bi...@ugov.gov> wrote:
> On Tuesday, January 3, 2012 11:51:19 AM, "Aaron Cordova" <aa...@cordovas.org> wrote:
> > Is it ok to disable the stringsummation aggregator and then enable the
> > summingcombiner through the shell? Should I take the table offline
> > first?
>
> If the table is quiet (i.e. not compacting), then it shouldn't be a problem to do the switch while it's online. Queries might get unexpected results during the switch. If you've removed the VersioningIterator from the table, then you could even make the change during compactions. However, if there is versioning, there would be the possibility of it removing versions instead of aggregating during compactions.
>
> Billie
>
>
Re: upgrade from 1.3 to 1.4
Posted by Adam Fuchs <ad...@ugov.gov>.
Oh yeah, and they're (mutually) idempotent too, right?
Adam
On Tue, Jan 3, 2012 at 12:43 PM, Adam Fuchs <ad...@ugov.gov> wrote:
> Why not add the combiner (so that both are enabled) and then remove the
> aggregator? The two iterators should be commutative and associative, so
> results should be reasonable and compactions should be safe when both are
> enabled in the tree.
>
> Adam
>
>
> On Tue, Jan 3, 2012 at 12:39 PM, Billie J Rinaldi <
> billie.j.rinaldi@ugov.gov> wrote:
>
>> On Tuesday, January 3, 2012 11:51:19 AM, "Aaron Cordova" <
>> aaron@cordovas.org> wrote:
>> > Is it ok to disable the stringsummation aggregator and then enable the
>> > summingcombiner through the shell? Should I take the table offline
>> > first?
>>
>> If the table is quiet (i.e. not compacting), then it shouldn't be a
>> problem to do the switch while it's online. Queries might get unexpected
>> results during the switch. If you've removed the VersioningIterator from
>> the table, then you could even make the change during compactions.
>> However, if there is versioning, there would be the possibility of it
>> removing versions instead of aggregating during compactions.
>>
>> Billie
>>
>
>
Re: upgrade from 1.3 to 1.4
Posted by Adam Fuchs <ad...@ugov.gov>.
Why not add the combiner (so that both are enabled) and then remove the
aggregator? The two iterators should be commutative and associative, so
results should be reasonable and compactions should be safe when both are
enabled in the tree.
Adam
On Tue, Jan 3, 2012 at 12:39 PM, Billie J Rinaldi <billie.j.rinaldi@ugov.gov
> wrote:
> On Tuesday, January 3, 2012 11:51:19 AM, "Aaron Cordova" <
> aaron@cordovas.org> wrote:
> > Is it ok to disable the stringsummation aggregator and then enable the
> > summingcombiner through the shell? Should I take the table offline
> > first?
>
> If the table is quiet (i.e. not compacting), then it shouldn't be a
> problem to do the switch while it's online. Queries might get unexpected
> results during the switch. If you've removed the VersioningIterator from
> the table, then you could even make the change during compactions.
> However, if there is versioning, there would be the possibility of it
> removing versions instead of aggregating during compactions.
>
> Billie
>
Re: upgrade from 1.3 to 1.4
Posted by Billie J Rinaldi <bi...@ugov.gov>.
On Tuesday, January 3, 2012 11:51:19 AM, "Aaron Cordova" <aa...@cordovas.org> wrote:
> Is it ok to disable the stringsummation aggregator and then enable the
> summingcombiner through the shell? Should I take the table offline
> first?
If the table is quiet (i.e. not compacting), then it shouldn't be a problem to do the switch while it's online. Queries might get unexpected results during the switch. If you've removed the VersioningIterator from the table, then you could even make the change during compactions. However, if there is versioning, there would be the possibility of it removing versions instead of aggregating during compactions.
Billie
Re: upgrade from 1.3 to 1.4
Posted by Aaron Cordova <aa...@cordovas.org>.
Is it ok to disable the stringsummation aggregator and then enable the summingcombiner through the shell? Should I take the table offline first?
On Jan 3, 2012, at 11:36 AM, Billie J Rinaldi wrote:
> On Tuesday, January 3, 2012 11:18:00 AM, "Aaron Cordova" <aa...@cordovas.org> wrote:
>> Also - if I have the StringSummation aggregator configured on a 1.3.5
>> table, when I bring up 1.4 will it continue to use that aggregator,
>> given that there's a new combiner framework? Is there anything I need
>> to do to keep it configured?
>
> Yes, that aggregator still exists in 1.4, but it is deprecated. I would recommend switching to the SummingCombiner at some point (org.apache.accumulo.core.iterators.user) with the "type" parameter set to "STRING".
>
> Billie
Re: upgrade from 1.3 to 1.4
Posted by Billie J Rinaldi <bi...@ugov.gov>.
On Tuesday, January 3, 2012 12:56:58 PM, "Aaron Cordova" <aa...@cordovas.org> wrote:
> Ahh that's awesome. So the function returns exactly one value for each
> key?
Yes, just like aggregators.
Billie
Re: upgrade from 1.3 to 1.4
Posted by Aaron Cordova <aa...@cordovas.org>.
Ahh that's awesome. So the function returns exactly one value for each key?
On Jan 3, 2012, at 12:53 PM, Billie J Rinaldi wrote:
> On Tuesday, January 3, 2012 11:56:50 AM, "Aaron Cordova" <aa...@cordovas.org> wrote:
>> BTW I love the label 'Combiner' instead of aggregator. I've been
>> describing their capabilities as equivalent to MapReduce combiners to
>> new Accumulo users, so this makes it even more clear.
>>
>> Are they easier to code up too? It'd be sweet if one could basically
>> just define a function such as
>>
>> public void combine(Key k, Iterator<Value> v)
>>
>> or some such simple signature
>
> That's basically it. Here's the method you implement:
> public Value reduce(Key k, Iterator<Value> v);
>
> We've also done something exciting with the TypedValueCombiner<V>. It uses an Encoder<V> which has decode and encode methods to translate from a Value's byte array to type V and back. Then you implement this method to do the combining:
> public V typedReduce(Key k, Iterator<V> v);
>
> We provide a LongCombiner which is a TypedValueCombiner<Long> with three built-in encoding types: STRING, VARLEN, and FIXEDLEN. Separating the encoding from the combining has made writing new combiners really easy. See the MaxCombiner and MinCombiner in addition to the SummingCombiner.
>
> Billie
Re: upgrade from 1.3 to 1.4
Posted by Billie J Rinaldi <bi...@ugov.gov>.
On Tuesday, January 3, 2012 11:56:50 AM, "Aaron Cordova" <aa...@cordovas.org> wrote:
> BTW I love the label 'Combiner' instead of aggregator. I've been
> describing their capabilities as equivalent to MapReduce combiners to
> new Accumulo users, so this makes it even more clear.
>
> Are they easier to code up too? It'd be sweet if one could basically
> just define a function such as
>
> public void combine(Key k, Iterator<Value> v)
>
> or some such simple signature
That's basically it. Here's the method you implement:
public Value reduce(Key k, Iterator<Value> v);
We've also done something exciting with the TypedValueCombiner<V>. It uses an Encoder<V> which has decode and encode methods to translate from a Value's byte array to type V and back. Then you implement this method to do the combining:
public V typedReduce(Key k, Iterator<V> v);
We provide a LongCombiner which is a TypedValueCombiner<Long> with three built-in encoding types: STRING, VARLEN, and FIXEDLEN. Separating the encoding from the combining has made writing new combiners really easy. See the MaxCombiner and MinCombiner in addition to the SummingCombiner.
Billie
Re: upgrade from 1.3 to 1.4
Posted by Aaron Cordova <aa...@cordovas.org>.
BTW I love the label 'Combiner' instead of aggregator. I've been describing their capabilities as equivalent to MapReduce combiners to new Accumulo users, so this makes it even more clear.
Are they easier to code up too? It'd be sweet if one could basically just define a function such as
public void combine(Key k, Iterator<Value> v)
or some such simple signature
On Jan 3, 2012, at 11:36 AM, Billie J Rinaldi wrote:
> On Tuesday, January 3, 2012 11:18:00 AM, "Aaron Cordova" <aa...@cordovas.org> wrote:
>> Also - if I have the StringSummation aggregator configured on a 1.3.5
>> table, when I bring up 1.4 will it continue to use that aggregator,
>> given that there's a new combiner framework? Is there anything I need
>> to do to keep it configured?
>
> Yes, that aggregator still exists in 1.4, but it is deprecated. I would recommend switching to the SummingCombiner at some point (org.apache.accumulo.core.iterators.user) with the "type" parameter set to "STRING".
>
> Billie
Re: upgrade from 1.3 to 1.4
Posted by Billie J Rinaldi <bi...@ugov.gov>.
On Tuesday, January 3, 2012 11:18:00 AM, "Aaron Cordova" <aa...@cordovas.org> wrote:
> Also - if I have the StringSummation aggregator configured on a 1.3.5
> table, when I bring up 1.4 will it continue to use that aggregator,
> given that there's a new combiner framework? Is there anything I need
> to do to keep it configured?
Yes, that aggregator still exists in 1.4, but it is deprecated. I would recommend switching to the SummingCombiner at some point (org.apache.accumulo.core.iterators.user) with the "type" parameter set to "STRING".
Billie
Re: upgrade from 1.3 to 1.4
Posted by Aaron Cordova <aa...@cordovas.org>.
Also - if I have the StringSummation aggregator configured on a 1.3.5 table, when I bring up 1.4 will it continue to use that aggregator, given that there's a new combiner framework? Is there anything I need to do to keep it configured?
On Jan 3, 2012, at 11:03 AM, Keith Turner wrote:
> It should happen automatically when the 1.4 code starts for the first
> time. So shut down 1.3 and start 1.4.
>
> On Mon, Jan 2, 2012 at 9:12 AM, Aaron Cordova <aa...@cordovas.org> wrote:
>> Is there anything that needs to be done to upgrade from 1.3 to 1.4?
Re: upgrade from 1.3 to 1.4
Posted by Adam Fuchs <ad...@ugov.gov>.
The 1.4 branch is pretty stable, although it's not as well tested as the
1.3 branch.
Adam
On Tue, Jan 3, 2012 at 11:04 AM, Aaron Cordova <aa...@cordovas.org> wrote:
> awesome. is the 1.4 branch stable? if not, is trunk stable?
>
> On Jan 3, 2012, at 11:03 AM, Keith Turner wrote:
>
> > It should happen automatically when the 1.4 code starts for the first
> > time. So shut down 1.3 and start 1.4.
> >
> > On Mon, Jan 2, 2012 at 9:12 AM, Aaron Cordova <aa...@cordovas.org>
> wrote:
> >> Is there anything that needs to be done to upgrade from 1.3 to 1.4?
>
>
Re: upgrade from 1.3 to 1.4
Posted by Aaron Cordova <aa...@cordovas.org>.
awesome. is the 1.4 branch stable? if not, is trunk stable?
On Jan 3, 2012, at 11:03 AM, Keith Turner wrote:
> It should happen automatically when the 1.4 code starts for the first
> time. So shut down 1.3 and start 1.4.
>
> On Mon, Jan 2, 2012 at 9:12 AM, Aaron Cordova <aa...@cordovas.org> wrote:
>> Is there anything that needs to be done to upgrade from 1.3 to 1.4?
Re: upgrade from 1.3 to 1.4
Posted by Keith Turner <ke...@deenlo.com>.
It should happen automatically when the 1.4 code starts for the first
time. So shut down 1.3 and start 1.4.
On Mon, Jan 2, 2012 at 9:12 AM, Aaron Cordova <aa...@cordovas.org> wrote:
> Is there anything that needs to be done to upgrade from 1.3 to 1.4?