Drill 1.1 and partition by

[James Jones <ja...@gmail.com>]

First I want to offer some congrats on the 1.1 release. We have be
hammering it with all of our regression tests and are seeing very good
results. Also I'm really appreciating the effort and content of the apache
drill documentation. Its much appreciated.

Our team is currently running several tests with the new Partition By
feature. One of our team members noticed store.partition.hash_distribute on
the sys.options page.

We noticed that it wasn't included here
https://drill.apache.org/docs/configuration-options-introduction/ and are
curious if that option is considered stable or just an over site with the
documentation.

We did find this post which describes the option well enough
http://mail-archives.apache.org/mod_mbox/drill-commits/201506.mbox/%3C38571170b14d484bba843f1a513befaf@git.apache.org%3E

Re: Drill 1.1 and partition by

[James Jones <ja...@gmail.com>]

Thanks I'll see if we can get some of CTAS timings that aren't partitioned
with and without the option enabled.


On Tue, Jul 7, 2015 at 2:56 PM, Steven Phillips <sp...@maprtech.com>
wrote:

> The feature was added late in the release cycle, and it wasn't tested as
> thoroughly as the default option. I think it should be perfectly ok to use;
> just be aware that it may lead to decreased performance when running CTAS
> operations.
> On the other hand, this could drastically reduce the number of files
> generated, which could have a positive effect on subsequent queries against
> the generated data.
>
> On Tue, Jul 7, 2015 at 8:03 AM, James Jones <ja...@gmail.com> wrote:
>
> > First I want to offer some congrats on the 1.1 release. We have be
> > hammering it with all of our regression tests and are seeing very good
> > results. Also I'm really appreciating the effort and content of the
> apache
> > drill documentation. Its much appreciated.
> >
> > Our team is currently running several tests with the new Partition By
> > feature. One of our team members noticed store.partition.hash_distribute
> on
> > the sys.options page.
> >
> > We noticed that it wasn't included here
> > https://drill.apache.org/docs/configuration-options-introduction/ and
> are
> > curious if that option is considered stable or just an over site with the
> > documentation.
> >
> > We did find this post which describes the option well enough
> >
> >
> http://mail-archives.apache.org/mod_mbox/drill-commits/201506.mbox/%3C38571170b14d484bba843f1a513befaf@git.apache.org%3E
> >
>
>
>
> --
>  Steven Phillips
>  Software Engineer
>
>  mapr.com
>

Re: Drill 1.1 and partition by

[Steven Phillips <sp...@maprtech.com>]

The feature was added late in the release cycle, and it wasn't tested as
thoroughly as the default option. I think it should be perfectly ok to use;
just be aware that it may lead to decreased performance when running CTAS
operations.
On the other hand, this could drastically reduce the number of files
generated, which could have a positive effect on subsequent queries against
the generated data.

On Tue, Jul 7, 2015 at 8:03 AM, James Jones <ja...@gmail.com> wrote:

> First I want to offer some congrats on the 1.1 release. We have be
> hammering it with all of our regression tests and are seeing very good
> results. Also I'm really appreciating the effort and content of the apache
> drill documentation. Its much appreciated.
>
> Our team is currently running several tests with the new Partition By
> feature. One of our team members noticed store.partition.hash_distribute on
> the sys.options page.
>
> We noticed that it wasn't included here
> https://drill.apache.org/docs/configuration-options-introduction/ and are
> curious if that option is considered stable or just an over site with the
> documentation.
>
> We did find this post which describes the option well enough
>
> http://mail-archives.apache.org/mod_mbox/drill-commits/201506.mbox/%3C38571170b14d484bba843f1a513befaf@git.apache.org%3E
>



-- 
 Steven Phillips
 Software Engineer

 mapr.com