Snappy as default table.file.compress.type?

[Josh Elser <jo...@gmail.com>]

Any reason we don't want to do this? Last rule-of-thumb I heard was that 
snappy is often close enough in compression to GZ but quite a bit faster 
(I don't remember exactly how much).

- Josh

Re: Snappy as default table.file.compress.type?

[Josh Elser <jo...@gmail.com>]

Ok, understood. Such a change would certainly require mention in release 
notes, user manual, etc.

Christopher wrote:
> Yes, it's a simple matter to install the dependency... it just might not be
> installed by default. I'd certainly recommend downstream vendors/packagers
> add it as a required or suggested dependency to their RPMs/DEBs/etc.,
> though.
>
> The snappy package on RHEL/CentOS provides libsnappy. The
> org.xerial.snappy:snappy-java dependency provides JNI support, but it looks
> like Hadoop doesn't use that and instead uses its own JNI stuffs. Neither
> seem to provide a non-native implementation, as far as I can tell. So, I
> guess I was wrong about that. You definitely need the native library
> installed for it to work at all.
>
> On Sat, Aug 13, 2016 at 11:42 PM Josh Elser<jo...@gmail.com>  wrote:
>
>> That's a fair point. I'm off in nebulous vendor land and tend to be removed
>> from pure Apache Hadoop artifacts. I feel like there's a snappy package (at
>> least on centos) which is enough, but understanding this would be good.
>>
>> Is there a nonnative snappy impl?
>>
>> On Aug 13, 2016 11:19 PM, "Christopher"<ct...@apache.org>  wrote:
>>
>>> Native libraries for snappy are also not typically installed by default
>> on
>>> Linux distros. Even if the hadoop native libraries are installed, the
>> user
>>> is likely going to end up using the Java implementation by default, I
>>> *think*, unless they take additional actions.
>>>
>>> On Sat, Aug 13, 2016 at 11:18 PM Adam Fuchs<af...@apache.org>  wrote:
>>>
>>>> In my experience gz gets roughly 1.5x to 2x better compression than
>>> snappy.
>>>> Snappy is definitely not a pareto improvement (although we tend to use
>>>> snappy by default). Since it's not always better I think you would
>> need a
>>>> more solid argument to change the default.
>>>>
>>>> Adam
>>>>
>>>> On Aug 13, 2016 8:06 PM, "Josh Elser"<jo...@gmail.com>  wrote:
>>>>
>>>>> Same motivation of using it as for making it the default. I am not
>>> aware
>>>>> of any downside to it. It's become pretty standard across all
>>>> installations
>>>>> I've worked with for years.
>>>>>
>>>>> Asking because I am no oracle on the matter. I could just be ignorant
>>> of
>>>>> some issue, but, given my current understanding, there is no downside
>>> for
>>>>> the average case.
>>>>>
>>>>> Christopher wrote:
>>>>>
>>>>>> Sorry. I wasn't clear. I understand the motivation for using it...
>> I'm
>>>>>> asking about the motivation for making it the default.
>>>>>>
>>>>>> Since both are available, I'm not sure the default matters *that*
>>> much,
>>>>>> but
>>>>>> it could be an unexpected change for those preferring GZ.
>>>>>>
>>>>>> Also, are there any risks regarding library availability of snappy?
>> GZ
>>>> is
>>>>>> pretty ubiquitous.
>>>>>>
>>>>>> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
>>>> wrote:
>>>>>> Uhh, besides what I already mentioned? (close in compressed size but
>>>>>>> "much" faster)
>>>>>>>
>>>>>>> Christopher wrote:
>>>>>>>
>>>>>>>> What's the motivation for changing it?
>>>>>>>>
>>>>>>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
>>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Any reason we don't want to do this? Last rule-of-thumb I heard
>> was
>>>> that
>>>>>>>>> snappy is often close enough in compression to GZ but quite a bit
>>>>>>>>> faster
>>>>>>>>> (I don't remember exactly how much).
>>>>>>>>>
>>>>>>>>> - Josh
>>>>>>>>>
>>>>>>>>>
>

Re: Snappy as default table.file.compress.type?

[Christopher <ct...@apache.org>]

Yes, it's a simple matter to install the dependency... it just might not be
installed by default. I'd certainly recommend downstream vendors/packagers
add it as a required or suggested dependency to their RPMs/DEBs/etc.,
though.

The snappy package on RHEL/CentOS provides libsnappy. The
org.xerial.snappy:snappy-java dependency provides JNI support, but it looks
like Hadoop doesn't use that and instead uses its own JNI stuffs. Neither
seem to provide a non-native implementation, as far as I can tell. So, I
guess I was wrong about that. You definitely need the native library
installed for it to work at all.

On Sat, Aug 13, 2016 at 11:42 PM Josh Elser <jo...@gmail.com> wrote:

> That's a fair point. I'm off in nebulous vendor land and tend to be removed
> from pure Apache Hadoop artifacts. I feel like there's a snappy package (at
> least on centos) which is enough, but understanding this would be good.
>
> Is there a nonnative snappy impl?
>
> On Aug 13, 2016 11:19 PM, "Christopher" <ct...@apache.org> wrote:
>
> > Native libraries for snappy are also not typically installed by default
> on
> > Linux distros. Even if the hadoop native libraries are installed, the
> user
> > is likely going to end up using the Java implementation by default, I
> > *think*, unless they take additional actions.
> >
> > On Sat, Aug 13, 2016 at 11:18 PM Adam Fuchs <af...@apache.org> wrote:
> >
> > > In my experience gz gets roughly 1.5x to 2x better compression than
> > snappy.
> > > Snappy is definitely not a pareto improvement (although we tend to use
> > > snappy by default). Since it's not always better I think you would
> need a
> > > more solid argument to change the default.
> > >
> > > Adam
> > >
> > > On Aug 13, 2016 8:06 PM, "Josh Elser" <jo...@gmail.com> wrote:
> > >
> > > > Same motivation of using it as for making it the default. I am not
> > aware
> > > > of any downside to it. It's become pretty standard across all
> > > installations
> > > > I've worked with for years.
> > > >
> > > > Asking because I am no oracle on the matter. I could just be ignorant
> > of
> > > > some issue, but, given my current understanding, there is no downside
> > for
> > > > the average case.
> > > >
> > > > Christopher wrote:
> > > >
> > > >> Sorry. I wasn't clear. I understand the motivation for using it...
> I'm
> > > >> asking about the motivation for making it the default.
> > > >>
> > > >> Since both are available, I'm not sure the default matters *that*
> > much,
> > > >> but
> > > >> it could be an unexpected change for those preferring GZ.
> > > >>
> > > >> Also, are there any risks regarding library availability of snappy?
> GZ
> > > is
> > > >> pretty ubiquitous.
> > > >>
> > > >> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
> > > wrote:
> > > >>
> > > >> Uhh, besides what I already mentioned? (close in compressed size but
> > > >>> "much" faster)
> > > >>>
> > > >>> Christopher wrote:
> > > >>>
> > > >>>> What's the motivation for changing it?
> > > >>>>
> > > >>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
> > > >>>>
> > > >>> wrote:
> > > >>>
> > > >>>> Any reason we don't want to do this? Last rule-of-thumb I heard
> was
> > > that
> > > >>>>> snappy is often close enough in compression to GZ but quite a bit
> > > >>>>> faster
> > > >>>>> (I don't remember exactly how much).
> > > >>>>>
> > > >>>>> - Josh
> > > >>>>>
> > > >>>>>
> > > >>
> > >
> >
>

Re: Snappy as default table.file.compress.type?

[Josh Elser <jo...@gmail.com>]

That's a fair point. I'm off in nebulous vendor land and tend to be removed
from pure Apache Hadoop artifacts. I feel like there's a snappy package (at
least on centos) which is enough, but understanding this would be good.

Is there a nonnative snappy impl?

On Aug 13, 2016 11:19 PM, "Christopher" <ct...@apache.org> wrote:

> Native libraries for snappy are also not typically installed by default on
> Linux distros. Even if the hadoop native libraries are installed, the user
> is likely going to end up using the Java implementation by default, I
> *think*, unless they take additional actions.
>
> On Sat, Aug 13, 2016 at 11:18 PM Adam Fuchs <af...@apache.org> wrote:
>
> > In my experience gz gets roughly 1.5x to 2x better compression than
> snappy.
> > Snappy is definitely not a pareto improvement (although we tend to use
> > snappy by default). Since it's not always better I think you would need a
> > more solid argument to change the default.
> >
> > Adam
> >
> > On Aug 13, 2016 8:06 PM, "Josh Elser" <jo...@gmail.com> wrote:
> >
> > > Same motivation of using it as for making it the default. I am not
> aware
> > > of any downside to it. It's become pretty standard across all
> > installations
> > > I've worked with for years.
> > >
> > > Asking because I am no oracle on the matter. I could just be ignorant
> of
> > > some issue, but, given my current understanding, there is no downside
> for
> > > the average case.
> > >
> > > Christopher wrote:
> > >
> > >> Sorry. I wasn't clear. I understand the motivation for using it... I'm
> > >> asking about the motivation for making it the default.
> > >>
> > >> Since both are available, I'm not sure the default matters *that*
> much,
> > >> but
> > >> it could be an unexpected change for those preferring GZ.
> > >>
> > >> Also, are there any risks regarding library availability of snappy? GZ
> > is
> > >> pretty ubiquitous.
> > >>
> > >> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
> > wrote:
> > >>
> > >> Uhh, besides what I already mentioned? (close in compressed size but
> > >>> "much" faster)
> > >>>
> > >>> Christopher wrote:
> > >>>
> > >>>> What's the motivation for changing it?
> > >>>>
> > >>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
> > >>>>
> > >>> wrote:
> > >>>
> > >>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
> > that
> > >>>>> snappy is often close enough in compression to GZ but quite a bit
> > >>>>> faster
> > >>>>> (I don't remember exactly how much).
> > >>>>>
> > >>>>> - Josh
> > >>>>>
> > >>>>>
> > >>
> >
>

Re: Snappy as default table.file.compress.type?

[Josh Elser <jo...@gmail.com>]

I think I like this idea. The configuration templates we have, along 
with bin/bootstrap_config.sh, would make this easily consumable, IMO.

I'm not sure how many users know about/use that script though. I know 
that I personally still copy out of conf/examples/3GB-native and modify 
to suit my whims at the moment.

Marc P. wrote:
> Perhaps there is a happy medium, though, by not necessarily defining
> example configurations by the size of your memory footprint, but instead by
> performance configuration? Snappy could be the default for those who want a
> faster but less space cognizant implementation. Christopher's concerns
> would be allayed, and perhaps those who try Accumulo may get better
> performance by using Snappy?
>
> On Sat, Aug 13, 2016 at 11:19 PM, Christopher<ct...@apache.org>  wrote:
>
>> Native libraries for snappy are also not typically installed by default on
>> Linux distros. Even if the hadoop native libraries are installed, the user
>> is likely going to end up using the Java implementation by default, I
>> *think*, unless they take additional actions.
>>
>> On Sat, Aug 13, 2016 at 11:18 PM Adam Fuchs<af...@apache.org>  wrote:
>>
>>> In my experience gz gets roughly 1.5x to 2x better compression than
>> snappy.
>>> Snappy is definitely not a pareto improvement (although we tend to use
>>> snappy by default). Since it's not always better I think you would need a
>>> more solid argument to change the default.
>>>
>>> Adam
>>>
>>> On Aug 13, 2016 8:06 PM, "Josh Elser"<jo...@gmail.com>  wrote:
>>>
>>>> Same motivation of using it as for making it the default. I am not
>> aware
>>>> of any downside to it. It's become pretty standard across all
>>> installations
>>>> I've worked with for years.
>>>>
>>>> Asking because I am no oracle on the matter. I could just be ignorant
>> of
>>>> some issue, but, given my current understanding, there is no downside
>> for
>>>> the average case.
>>>>
>>>> Christopher wrote:
>>>>
>>>>> Sorry. I wasn't clear. I understand the motivation for using it... I'm
>>>>> asking about the motivation for making it the default.
>>>>>
>>>>> Since both are available, I'm not sure the default matters *that*
>> much,
>>>>> but
>>>>> it could be an unexpected change for those preferring GZ.
>>>>>
>>>>> Also, are there any risks regarding library availability of snappy? GZ
>>> is
>>>>> pretty ubiquitous.
>>>>>
>>>>> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
>>> wrote:
>>>>> Uhh, besides what I already mentioned? (close in compressed size but
>>>>>> "much" faster)
>>>>>>
>>>>>> Christopher wrote:
>>>>>>
>>>>>>> What's the motivation for changing it?
>>>>>>>
>>>>>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
>>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
>>> that
>>>>>>>> snappy is often close enough in compression to GZ but quite a bit
>>>>>>>> faster
>>>>>>>> (I don't remember exactly how much).
>>>>>>>>
>>>>>>>> - Josh
>>>>>>>>
>>>>>>>>
>

Re: Snappy as default table.file.compress.type?

["Marc P." <ma...@gmail.com>]

Perhaps there is a happy medium, though, by not necessarily defining
example configurations by the size of your memory footprint, but instead by
performance configuration? Snappy could be the default for those who want a
faster but less space cognizant implementation. Christopher's concerns
would be allayed, and perhaps those who try Accumulo may get better
performance by using Snappy?

On Sat, Aug 13, 2016 at 11:19 PM, Christopher <ct...@apache.org> wrote:

> Native libraries for snappy are also not typically installed by default on
> Linux distros. Even if the hadoop native libraries are installed, the user
> is likely going to end up using the Java implementation by default, I
> *think*, unless they take additional actions.
>
> On Sat, Aug 13, 2016 at 11:18 PM Adam Fuchs <af...@apache.org> wrote:
>
> > In my experience gz gets roughly 1.5x to 2x better compression than
> snappy.
> > Snappy is definitely not a pareto improvement (although we tend to use
> > snappy by default). Since it's not always better I think you would need a
> > more solid argument to change the default.
> >
> > Adam
> >
> > On Aug 13, 2016 8:06 PM, "Josh Elser" <jo...@gmail.com> wrote:
> >
> > > Same motivation of using it as for making it the default. I am not
> aware
> > > of any downside to it. It's become pretty standard across all
> > installations
> > > I've worked with for years.
> > >
> > > Asking because I am no oracle on the matter. I could just be ignorant
> of
> > > some issue, but, given my current understanding, there is no downside
> for
> > > the average case.
> > >
> > > Christopher wrote:
> > >
> > >> Sorry. I wasn't clear. I understand the motivation for using it... I'm
> > >> asking about the motivation for making it the default.
> > >>
> > >> Since both are available, I'm not sure the default matters *that*
> much,
> > >> but
> > >> it could be an unexpected change for those preferring GZ.
> > >>
> > >> Also, are there any risks regarding library availability of snappy? GZ
> > is
> > >> pretty ubiquitous.
> > >>
> > >> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
> > wrote:
> > >>
> > >> Uhh, besides what I already mentioned? (close in compressed size but
> > >>> "much" faster)
> > >>>
> > >>> Christopher wrote:
> > >>>
> > >>>> What's the motivation for changing it?
> > >>>>
> > >>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
> > >>>>
> > >>> wrote:
> > >>>
> > >>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
> > that
> > >>>>> snappy is often close enough in compression to GZ but quite a bit
> > >>>>> faster
> > >>>>> (I don't remember exactly how much).
> > >>>>>
> > >>>>> - Josh
> > >>>>>
> > >>>>>
> > >>
> >
>

Re: Snappy as default table.file.compress.type?

[Christopher <ct...@apache.org>]

Native libraries for snappy are also not typically installed by default on
Linux distros. Even if the hadoop native libraries are installed, the user
is likely going to end up using the Java implementation by default, I
*think*, unless they take additional actions.

On Sat, Aug 13, 2016 at 11:18 PM Adam Fuchs <af...@apache.org> wrote:

> In my experience gz gets roughly 1.5x to 2x better compression than snappy.
> Snappy is definitely not a pareto improvement (although we tend to use
> snappy by default). Since it's not always better I think you would need a
> more solid argument to change the default.
>
> Adam
>
> On Aug 13, 2016 8:06 PM, "Josh Elser" <jo...@gmail.com> wrote:
>
> > Same motivation of using it as for making it the default. I am not aware
> > of any downside to it. It's become pretty standard across all
> installations
> > I've worked with for years.
> >
> > Asking because I am no oracle on the matter. I could just be ignorant of
> > some issue, but, given my current understanding, there is no downside for
> > the average case.
> >
> > Christopher wrote:
> >
> >> Sorry. I wasn't clear. I understand the motivation for using it... I'm
> >> asking about the motivation for making it the default.
> >>
> >> Since both are available, I'm not sure the default matters *that* much,
> >> but
> >> it could be an unexpected change for those preferring GZ.
> >>
> >> Also, are there any risks regarding library availability of snappy? GZ
> is
> >> pretty ubiquitous.
> >>
> >> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
> wrote:
> >>
> >> Uhh, besides what I already mentioned? (close in compressed size but
> >>> "much" faster)
> >>>
> >>> Christopher wrote:
> >>>
> >>>> What's the motivation for changing it?
> >>>>
> >>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
> >>>>
> >>> wrote:
> >>>
> >>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
> that
> >>>>> snappy is often close enough in compression to GZ but quite a bit
> >>>>> faster
> >>>>> (I don't remember exactly how much).
> >>>>>
> >>>>> - Josh
> >>>>>
> >>>>>
> >>
>

Re: Snappy as default table.file.compress.type?

[Michael Wall <mj...@gmail.com>]

I like the idea of making snappy the default.  However, I am concerned
about raising the barrier of entry to new users by adding yet another
dependency to install.

On Mon, Aug 15, 2016 at 11:13 AM, Josh Elser <jo...@gmail.com> wrote:

> No, I never asserted that Snappy is *always* the better choice. I would
> say that I believe Snappy is better in *most cases*.
>
> Most users I talk to (with and without Accumulo involved) have plenty of
> disk space available to them. It is rare that space on disk is actually a
> concern. Instead, performance is usually the primary metric of concern. To
> be crystal clear, this is only my opinion on users I've talked to, not an
> assertion on everyone.
>
> I do not believe I need a better argument than "on average, we can make
> out of the box performance better for most users". I suppose we'll have to
> disagree on that point. Thanks for clarifying your opinions on the topic.
>
>
> Adam Fuchs wrote:
>
>> If the crux of your argument was that snappy is always a better choice,
>> then my retort was to say it is not, since sometimes compression ratio can
>> be a dominant factor. Changes to defaults are disruptive for existing
>> users, so you need a better argument. I don't mean that you shouldn't
>> continue to debate the merits. By all means, do continue the conversation.
>>
>> Adam
>>
>> On Aug 13, 2016 8:39 PM, "Josh Elser"<jo...@gmail.com>  wrote:
>>
>>> Your argument fails to address the performance benefits. I could pose the
>>> same question back to you: you need to prove why we shouldn't use the
>>> faster compression algorithm.
>>>
>>> I don't mean to be snarky, but your argument is shutting down
>>>
>> conversation.
>>
>>> I appreciate you sharing the opinion but don't feel like it's encouraging
>>> discussion.
>>>
>>> On Aug 13, 2016 11:18 PM, "Adam Fuchs"<af...@apache.org>  wrote:
>>>
>>> In my experience gz gets roughly 1.5x to 2x better compression than
>>>>
>>> snappy.
>>
>>> Snappy is definitely not a pareto improvement (although we tend to use
>>>> snappy by default). Since it's not always better I think you would need
>>>>
>>> a
>>
>>> more solid argument to change the default.
>>>>
>>>> Adam
>>>>
>>>> On Aug 13, 2016 8:06 PM, "Josh Elser"<jo...@gmail.com>  wrote:
>>>>
>>>> Same motivation of using it as for making it the default. I am not
>>>>>
>>>> aware
>>
>>> of any downside to it. It's become pretty standard across all
>>>>>
>>>> installations
>>>>
>>>>> I've worked with for years.
>>>>>
>>>>> Asking because I am no oracle on the matter. I could just be ignorant
>>>>>
>>>> of
>>
>>> some issue, but, given my current understanding, there is no downside
>>>>>
>>>> for
>>
>>> the average case.
>>>>>
>>>>> Christopher wrote:
>>>>>
>>>>> Sorry. I wasn't clear. I understand the motivation for using it...
>>>>>>
>>>>> I'm
>>
>>> asking about the motivation for making it the default.
>>>>>>
>>>>>> Since both are available, I'm not sure the default matters *that*
>>>>>>
>>>>> much,
>>
>>> but
>>>>>> it could be an unexpected change for those preferring GZ.
>>>>>>
>>>>>> Also, are there any risks regarding library availability of snappy?
>>>>>>
>>>>> GZ
>>
>>> is
>>>>
>>>>> pretty ubiquitous.
>>>>>>
>>>>>> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
>>>>>>
>>>>> wrote:
>>>>
>>>>> Uhh, besides what I already mentioned? (close in compressed size but
>>>>>>
>>>>>>> "much" faster)
>>>>>>>
>>>>>>> Christopher wrote:
>>>>>>>
>>>>>>> What's the motivation for changing it?
>>>>>>>>
>>>>>>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>
>>>>>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
>>>>>>>>
>>>>>>> that
>>>>
>>>>> snappy is often close enough in compression to GZ but quite a bit
>>>>>>>>> faster
>>>>>>>>> (I don't remember exactly how much).
>>>>>>>>>
>>>>>>>>> - Josh
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>

Re: Snappy as default table.file.compress.type?

[Josh Elser <jo...@gmail.com>]

I agree it would be disruptive in the case that you outlined. This is 
why we have release notes and semver, though.

I think this change should only go into a major release for downstream 
stability. Even though how Accumulo creates and manages files is not 
covered by our compatibility statement (it could), I don't feel like 
it's worth trying to shoe-horn such a change into a minor release.

Adam Fuchs wrote:
> We need to consider the scenario in which somebody has written an
> application on Accumulo that uses the default compression codec. If we
> change the default, their app's behavior will change when they upgrade
> Accumulo, either because an existing table will start using snappy or
> because their app creates new tables and those new tables will start using
> snappy. This change could be disruptive, especially in the case that the
> application developers have moved on to other projects and are no longer
> available to fix the app. This is why I think you need a stronger argument
> than snappy being better on average than gzip. I agree that snappy is
> better on average, and even in most cases, but I'm not convinced we should
> change the default.
>
> Adam
>
>
> On Aug 15, 2016 8:14 AM, "Josh Elser"<jo...@gmail.com>  wrote:
>
>> No, I never asserted that Snappy is *always* the better choice. I would
>> say that I believe Snappy is better in *most cases*.
>>
>> Most users I talk to (with and without Accumulo involved) have plenty of
>> disk space available to them. It is rare that space on disk is actually a
>> concern. Instead, performance is usually the primary metric of concern. To
>> be crystal clear, this is only my opinion on users I've talked to, not an
>> assertion on everyone.
>>
>> I do not believe I need a better argument than "on average, we can make
>> out of the box performance better for most users". I suppose we'll have to
>> disagree on that point. Thanks for clarifying your opinions on the topic.
>>
>> Adam Fuchs wrote:
>>
>>> If the crux of your argument was that snappy is always a better choice,
>>> then my retort was to say it is not, since sometimes compression ratio can
>>> be a dominant factor. Changes to defaults are disruptive for existing
>>> users, so you need a better argument. I don't mean that you shouldn't
>>> continue to debate the merits. By all means, do continue the conversation.
>>>
>>> Adam
>>>
>>> On Aug 13, 2016 8:39 PM, "Josh Elser"<jo...@gmail.com>   wrote:
>>>
>>>> Your argument fails to address the performance benefits. I could pose the
>>>> same question back to you: you need to prove why we shouldn't use the
>>>> faster compression algorithm.
>>>>
>>>> I don't mean to be snarky, but your argument is shutting down
>>>>
>>> conversation.
>>>
>>>> I appreciate you sharing the opinion but don't feel like it's encouraging
>>>> discussion.
>>>>
>>>> On Aug 13, 2016 11:18 PM, "Adam Fuchs"<af...@apache.org>   wrote:
>>>>
>>>> In my experience gz gets roughly 1.5x to 2x better compression than
>>>> snappy.
>>>> Snappy is definitely not a pareto improvement (although we tend to use
>>>>> snappy by default). Since it's not always better I think you would need
>>>>>
>>>> a
>>>> more solid argument to change the default.
>>>>> Adam
>>>>>
>>>>> On Aug 13, 2016 8:06 PM, "Josh Elser"<jo...@gmail.com>   wrote:
>>>>>
>>>>> Same motivation of using it as for making it the default. I am not
>>>>> aware
>>>> of any downside to it. It's become pretty standard across all
>>>>> installations
>>>>>
>>>>>> I've worked with for years.
>>>>>>
>>>>>> Asking because I am no oracle on the matter. I could just be ignorant
>>>>>>
>>>>> of
>>>> some issue, but, given my current understanding, there is no downside
>>>>> for
>>>> the average case.
>>>>>> Christopher wrote:
>>>>>>
>>>>>> Sorry. I wasn't clear. I understand the motivation for using it...
>>>>>> I'm
>>>> asking about the motivation for making it the default.
>>>>>>> Since both are available, I'm not sure the default matters *that*
>>>>>>>
>>>>>> much,
>>>> but
>>>>>>> it could be an unexpected change for those preferring GZ.
>>>>>>>
>>>>>>> Also, are there any risks regarding library availability of snappy?
>>>>>>>
>>>>>> GZ
>>>> is
>>>>>> pretty ubiquitous.
>>>>>>> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
>>>>>>>
>>>>>> wrote:
>>>>>> Uhh, besides what I already mentioned? (close in compressed size but
>>>>>>>> "much" faster)
>>>>>>>>
>>>>>>>> Christopher wrote:
>>>>>>>>
>>>>>>>> What's the motivation for changing it?
>>>>>>>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
>>>>>>>> that
>>>>>> snappy is often close enough in compression to GZ but quite a bit
>>>>>>>>>> faster
>>>>>>>>>> (I don't remember exactly how much).
>>>>>>>>>>
>>>>>>>>>> - Josh
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>

Re: Snappy as default table.file.compress.type?

[Adam Fuchs <af...@apache.org>]

We need to consider the scenario in which somebody has written an
application on Accumulo that uses the default compression codec. If we
change the default, their app's behavior will change when they upgrade
Accumulo, either because an existing table will start using snappy or
because their app creates new tables and those new tables will start using
snappy. This change could be disruptive, especially in the case that the
application developers have moved on to other projects and are no longer
available to fix the app. This is why I think you need a stronger argument
than snappy being better on average than gzip. I agree that snappy is
better on average, and even in most cases, but I'm not convinced we should
change the default.

Adam


On Aug 15, 2016 8:14 AM, "Josh Elser" <jo...@gmail.com> wrote:

> No, I never asserted that Snappy is *always* the better choice. I would
> say that I believe Snappy is better in *most cases*.
>
> Most users I talk to (with and without Accumulo involved) have plenty of
> disk space available to them. It is rare that space on disk is actually a
> concern. Instead, performance is usually the primary metric of concern. To
> be crystal clear, this is only my opinion on users I've talked to, not an
> assertion on everyone.
>
> I do not believe I need a better argument than "on average, we can make
> out of the box performance better for most users". I suppose we'll have to
> disagree on that point. Thanks for clarifying your opinions on the topic.
>
> Adam Fuchs wrote:
>
>> If the crux of your argument was that snappy is always a better choice,
>> then my retort was to say it is not, since sometimes compression ratio can
>> be a dominant factor. Changes to defaults are disruptive for existing
>> users, so you need a better argument. I don't mean that you shouldn't
>> continue to debate the merits. By all means, do continue the conversation.
>>
>> Adam
>>
>> On Aug 13, 2016 8:39 PM, "Josh Elser"<jo...@gmail.com>  wrote:
>>
>>> Your argument fails to address the performance benefits. I could pose the
>>> same question back to you: you need to prove why we shouldn't use the
>>> faster compression algorithm.
>>>
>>> I don't mean to be snarky, but your argument is shutting down
>>>
>> conversation.
>>
>>> I appreciate you sharing the opinion but don't feel like it's encouraging
>>> discussion.
>>>
>>> On Aug 13, 2016 11:18 PM, "Adam Fuchs"<af...@apache.org>  wrote:
>>>
>>> In my experience gz gets roughly 1.5x to 2x better compression than
>>>>
>>> snappy.
>>
>>> Snappy is definitely not a pareto improvement (although we tend to use
>>>> snappy by default). Since it's not always better I think you would need
>>>>
>>> a
>>
>>> more solid argument to change the default.
>>>>
>>>> Adam
>>>>
>>>> On Aug 13, 2016 8:06 PM, "Josh Elser"<jo...@gmail.com>  wrote:
>>>>
>>>> Same motivation of using it as for making it the default. I am not
>>>>>
>>>> aware
>>
>>> of any downside to it. It's become pretty standard across all
>>>>>
>>>> installations
>>>>
>>>>> I've worked with for years.
>>>>>
>>>>> Asking because I am no oracle on the matter. I could just be ignorant
>>>>>
>>>> of
>>
>>> some issue, but, given my current understanding, there is no downside
>>>>>
>>>> for
>>
>>> the average case.
>>>>>
>>>>> Christopher wrote:
>>>>>
>>>>> Sorry. I wasn't clear. I understand the motivation for using it...
>>>>>>
>>>>> I'm
>>
>>> asking about the motivation for making it the default.
>>>>>>
>>>>>> Since both are available, I'm not sure the default matters *that*
>>>>>>
>>>>> much,
>>
>>> but
>>>>>> it could be an unexpected change for those preferring GZ.
>>>>>>
>>>>>> Also, are there any risks regarding library availability of snappy?
>>>>>>
>>>>> GZ
>>
>>> is
>>>>
>>>>> pretty ubiquitous.
>>>>>>
>>>>>> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
>>>>>>
>>>>> wrote:
>>>>
>>>>> Uhh, besides what I already mentioned? (close in compressed size but
>>>>>>
>>>>>>> "much" faster)
>>>>>>>
>>>>>>> Christopher wrote:
>>>>>>>
>>>>>>> What's the motivation for changing it?
>>>>>>>>
>>>>>>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>
>>>>>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
>>>>>>>>
>>>>>>> that
>>>>
>>>>> snappy is often close enough in compression to GZ but quite a bit
>>>>>>>>> faster
>>>>>>>>> (I don't remember exactly how much).
>>>>>>>>>
>>>>>>>>> - Josh
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>

Re: Snappy as default table.file.compress.type?

[Josh Elser <jo...@gmail.com>]

No, I never asserted that Snappy is *always* the better choice. I would 
say that I believe Snappy is better in *most cases*.

Most users I talk to (with and without Accumulo involved) have plenty of 
disk space available to them. It is rare that space on disk is actually 
a concern. Instead, performance is usually the primary metric of 
concern. To be crystal clear, this is only my opinion on users I've 
talked to, not an assertion on everyone.

I do not believe I need a better argument than "on average, we can make 
out of the box performance better for most users". I suppose we'll have 
to disagree on that point. Thanks for clarifying your opinions on the topic.

Adam Fuchs wrote:
> If the crux of your argument was that snappy is always a better choice,
> then my retort was to say it is not, since sometimes compression ratio can
> be a dominant factor. Changes to defaults are disruptive for existing
> users, so you need a better argument. I don't mean that you shouldn't
> continue to debate the merits. By all means, do continue the conversation.
>
> Adam
>
> On Aug 13, 2016 8:39 PM, "Josh Elser"<jo...@gmail.com>  wrote:
>> Your argument fails to address the performance benefits. I could pose the
>> same question back to you: you need to prove why we shouldn't use the
>> faster compression algorithm.
>>
>> I don't mean to be snarky, but your argument is shutting down
> conversation.
>> I appreciate you sharing the opinion but don't feel like it's encouraging
>> discussion.
>>
>> On Aug 13, 2016 11:18 PM, "Adam Fuchs"<af...@apache.org>  wrote:
>>
>>> In my experience gz gets roughly 1.5x to 2x better compression than
> snappy.
>>> Snappy is definitely not a pareto improvement (although we tend to use
>>> snappy by default). Since it's not always better I think you would need
> a
>>> more solid argument to change the default.
>>>
>>> Adam
>>>
>>> On Aug 13, 2016 8:06 PM, "Josh Elser"<jo...@gmail.com>  wrote:
>>>
>>>> Same motivation of using it as for making it the default. I am not
> aware
>>>> of any downside to it. It's become pretty standard across all
>>> installations
>>>> I've worked with for years.
>>>>
>>>> Asking because I am no oracle on the matter. I could just be ignorant
> of
>>>> some issue, but, given my current understanding, there is no downside
> for
>>>> the average case.
>>>>
>>>> Christopher wrote:
>>>>
>>>>> Sorry. I wasn't clear. I understand the motivation for using it...
> I'm
>>>>> asking about the motivation for making it the default.
>>>>>
>>>>> Since both are available, I'm not sure the default matters *that*
> much,
>>>>> but
>>>>> it could be an unexpected change for those preferring GZ.
>>>>>
>>>>> Also, are there any risks regarding library availability of snappy?
> GZ
>>> is
>>>>> pretty ubiquitous.
>>>>>
>>>>> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
>>> wrote:
>>>>> Uhh, besides what I already mentioned? (close in compressed size but
>>>>>> "much" faster)
>>>>>>
>>>>>> Christopher wrote:
>>>>>>
>>>>>>> What's the motivation for changing it?
>>>>>>>
>>>>>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
>>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
>>> that
>>>>>>>> snappy is often close enough in compression to GZ but quite a bit
>>>>>>>> faster
>>>>>>>> (I don't remember exactly how much).
>>>>>>>>
>>>>>>>> - Josh
>>>>>>>>
>>>>>>>>
>

Re: Snappy as default table.file.compress.type?

[Adam Fuchs <af...@apache.org>]

If the crux of your argument was that snappy is always a better choice,
then my retort was to say it is not, since sometimes compression ratio can
be a dominant factor. Changes to defaults are disruptive for existing
users, so you need a better argument. I don't mean that you shouldn't
continue to debate the merits. By all means, do continue the conversation.

Adam

On Aug 13, 2016 8:39 PM, "Josh Elser" <jo...@gmail.com> wrote:
>
> Your argument fails to address the performance benefits. I could pose the
> same question back to you: you need to prove why we shouldn't use the
> faster compression algorithm.
>
> I don't mean to be snarky, but your argument is shutting down
conversation.
> I appreciate you sharing the opinion but don't feel like it's encouraging
> discussion.
>
> On Aug 13, 2016 11:18 PM, "Adam Fuchs" <af...@apache.org> wrote:
>
> > In my experience gz gets roughly 1.5x to 2x better compression than
snappy.
> > Snappy is definitely not a pareto improvement (although we tend to use
> > snappy by default). Since it's not always better I think you would need
a
> > more solid argument to change the default.
> >
> > Adam
> >
> > On Aug 13, 2016 8:06 PM, "Josh Elser" <jo...@gmail.com> wrote:
> >
> > > Same motivation of using it as for making it the default. I am not
aware
> > > of any downside to it. It's become pretty standard across all
> > installations
> > > I've worked with for years.
> > >
> > > Asking because I am no oracle on the matter. I could just be ignorant
of
> > > some issue, but, given my current understanding, there is no downside
for
> > > the average case.
> > >
> > > Christopher wrote:
> > >
> > >> Sorry. I wasn't clear. I understand the motivation for using it...
I'm
> > >> asking about the motivation for making it the default.
> > >>
> > >> Since both are available, I'm not sure the default matters *that*
much,
> > >> but
> > >> it could be an unexpected change for those preferring GZ.
> > >>
> > >> Also, are there any risks regarding library availability of snappy?
GZ
> > is
> > >> pretty ubiquitous.
> > >>
> > >> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
> > wrote:
> > >>
> > >> Uhh, besides what I already mentioned? (close in compressed size but
> > >>> "much" faster)
> > >>>
> > >>> Christopher wrote:
> > >>>
> > >>>> What's the motivation for changing it?
> > >>>>
> > >>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
> > >>>>
> > >>> wrote:
> > >>>
> > >>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
> > that
> > >>>>> snappy is often close enough in compression to GZ but quite a bit
> > >>>>> faster
> > >>>>> (I don't remember exactly how much).
> > >>>>>
> > >>>>> - Josh
> > >>>>>
> > >>>>>
> > >>
> >

Re: Snappy as default table.file.compress.type?

[Josh Elser <jo...@gmail.com>]

Your argument fails to address the performance benefits. I could pose the
same question back to you: you need to prove why we shouldn't use the
faster compression algorithm.

I don't mean to be snarky, but your argument is shutting down conversation.
I appreciate you sharing the opinion but don't feel like it's encouraging
discussion.

On Aug 13, 2016 11:18 PM, "Adam Fuchs" <af...@apache.org> wrote:

> In my experience gz gets roughly 1.5x to 2x better compression than snappy.
> Snappy is definitely not a pareto improvement (although we tend to use
> snappy by default). Since it's not always better I think you would need a
> more solid argument to change the default.
>
> Adam
>
> On Aug 13, 2016 8:06 PM, "Josh Elser" <jo...@gmail.com> wrote:
>
> > Same motivation of using it as for making it the default. I am not aware
> > of any downside to it. It's become pretty standard across all
> installations
> > I've worked with for years.
> >
> > Asking because I am no oracle on the matter. I could just be ignorant of
> > some issue, but, given my current understanding, there is no downside for
> > the average case.
> >
> > Christopher wrote:
> >
> >> Sorry. I wasn't clear. I understand the motivation for using it... I'm
> >> asking about the motivation for making it the default.
> >>
> >> Since both are available, I'm not sure the default matters *that* much,
> >> but
> >> it could be an unexpected change for those preferring GZ.
> >>
> >> Also, are there any risks regarding library availability of snappy? GZ
> is
> >> pretty ubiquitous.
> >>
> >> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>
> wrote:
> >>
> >> Uhh, besides what I already mentioned? (close in compressed size but
> >>> "much" faster)
> >>>
> >>> Christopher wrote:
> >>>
> >>>> What's the motivation for changing it?
> >>>>
> >>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
> >>>>
> >>> wrote:
> >>>
> >>>> Any reason we don't want to do this? Last rule-of-thumb I heard was
> that
> >>>>> snappy is often close enough in compression to GZ but quite a bit
> >>>>> faster
> >>>>> (I don't remember exactly how much).
> >>>>>
> >>>>> - Josh
> >>>>>
> >>>>>
> >>
>

Re: Snappy as default table.file.compress.type?

[Adam Fuchs <af...@apache.org>]

In my experience gz gets roughly 1.5x to 2x better compression than snappy.
Snappy is definitely not a pareto improvement (although we tend to use
snappy by default). Since it's not always better I think you would need a
more solid argument to change the default.

Adam

On Aug 13, 2016 8:06 PM, "Josh Elser" <jo...@gmail.com> wrote:

> Same motivation of using it as for making it the default. I am not aware
> of any downside to it. It's become pretty standard across all installations
> I've worked with for years.
>
> Asking because I am no oracle on the matter. I could just be ignorant of
> some issue, but, given my current understanding, there is no downside for
> the average case.
>
> Christopher wrote:
>
>> Sorry. I wasn't clear. I understand the motivation for using it... I'm
>> asking about the motivation for making it the default.
>>
>> Since both are available, I'm not sure the default matters *that* much,
>> but
>> it could be an unexpected change for those preferring GZ.
>>
>> Also, are there any risks regarding library availability of snappy? GZ is
>> pretty ubiquitous.
>>
>> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>  wrote:
>>
>> Uhh, besides what I already mentioned? (close in compressed size but
>>> "much" faster)
>>>
>>> Christopher wrote:
>>>
>>>> What's the motivation for changing it?
>>>>
>>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
>>>>
>>> wrote:
>>>
>>>> Any reason we don't want to do this? Last rule-of-thumb I heard was that
>>>>> snappy is often close enough in compression to GZ but quite a bit
>>>>> faster
>>>>> (I don't remember exactly how much).
>>>>>
>>>>> - Josh
>>>>>
>>>>>
>>

Re: Snappy as default table.file.compress.type?

[Josh Elser <jo...@gmail.com>]

Same motivation of using it as for making it the default. I am not aware 
of any downside to it. It's become pretty standard across all 
installations I've worked with for years.

Asking because I am no oracle on the matter. I could just be ignorant of 
some issue, but, given my current understanding, there is no downside 
for the average case.

Christopher wrote:
> Sorry. I wasn't clear. I understand the motivation for using it... I'm
> asking about the motivation for making it the default.
>
> Since both are available, I'm not sure the default matters *that* much, but
> it could be an unexpected change for those preferring GZ.
>
> Also, are there any risks regarding library availability of snappy? GZ is
> pretty ubiquitous.
>
> On Sat, Aug 13, 2016 at 10:59 PM Josh Elser<jo...@gmail.com>  wrote:
>
>> Uhh, besides what I already mentioned? (close in compressed size but
>> "much" faster)
>>
>> Christopher wrote:
>>> What's the motivation for changing it?
>>>
>>> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
>> wrote:
>>>> Any reason we don't want to do this? Last rule-of-thumb I heard was that
>>>> snappy is often close enough in compression to GZ but quite a bit faster
>>>> (I don't remember exactly how much).
>>>>
>>>> - Josh
>>>>
>

Re: Snappy as default table.file.compress.type?

[Christopher <ct...@apache.org>]

Sorry. I wasn't clear. I understand the motivation for using it... I'm
asking about the motivation for making it the default.

Since both are available, I'm not sure the default matters *that* much, but
it could be an unexpected change for those preferring GZ.

Also, are there any risks regarding library availability of snappy? GZ is
pretty ubiquitous.

On Sat, Aug 13, 2016 at 10:59 PM Josh Elser <jo...@gmail.com> wrote:

> Uhh, besides what I already mentioned? (close in compressed size but
> "much" faster)
>
> Christopher wrote:
> > What's the motivation for changing it?
> >
> > On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>
> wrote:
> >
> >> Any reason we don't want to do this? Last rule-of-thumb I heard was that
> >> snappy is often close enough in compression to GZ but quite a bit faster
> >> (I don't remember exactly how much).
> >>
> >> - Josh
> >>
> >
>

Re: Snappy as default table.file.compress.type?

[Josh Elser <jo...@gmail.com>]

Uhh, besides what I already mentioned? (close in compressed size but 
"much" faster)

Christopher wrote:
> What's the motivation for changing it?
>
> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser<jo...@gmail.com>  wrote:
>
>> Any reason we don't want to do this? Last rule-of-thumb I heard was that
>> snappy is often close enough in compression to GZ but quite a bit faster
>> (I don't remember exactly how much).
>>
>> - Josh
>>
>

Re: Snappy as default table.file.compress.type?

["Marc P." <ma...@gmail.com>]

In many cases , I would imagine faster is better .

On Sat, Aug 13, 2016, 10:56 PM Christopher <ct...@apache.org> wrote:

> What's the motivation for changing it?
>
> On Sat, Aug 13, 2016 at 10:47 PM Josh Elser <jo...@gmail.com> wrote:
>
> > Any reason we don't want to do this? Last rule-of-thumb I heard was that
> > snappy is often close enough in compression to GZ but quite a bit faster
> > (I don't remember exactly how much).
> >
> > - Josh
> >
>

Re: Snappy as default table.file.compress.type?

[Christopher <ct...@apache.org>]

What's the motivation for changing it?

On Sat, Aug 13, 2016 at 10:47 PM Josh Elser <jo...@gmail.com> wrote:

> Any reason we don't want to do this? Last rule-of-thumb I heard was that
> snappy is often close enough in compression to GZ but quite a bit faster
> (I don't remember exactly how much).
>
> - Josh
>