You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Dechang Gu <dg...@maprtech.com> on 2015/11/18 20:53:11 UTC
Drill tests on HPC
Hi All,
With the help and support from Edmon, I am going to leverage the
HPC resources at NICS ( https://www.nics.tennessee.edu/ <https://www.nics.tennessee.edu/>) for some Drill tests.
The goal is to explore how Drill performs on HPC platform.
1. Platform configuration: details can be found here:
https://www.nics.tennessee.edu/computing-resources/beacon/configuration <https://www.nics.tennessee.edu/computing-resources/beacon/configuration>
2. Tests: TPCH and/or TPCDS (subset of queries)
datasize: will start with SF1000 (~1TB), may increase to larger dataset (depending on storage space available)
test runs: 1 thread (like power run)
multiple threads (like Throughput run)
3. Set up:
— Drill (master) on Apache Hadoop
— Drill (master) on MapR-FS
Any comments/suggestions?
Thanks,
Dechang
Re: Drill tests on HPC
Posted by Dechang Gu <dg...@maprtech.com>.
Sure. I think the default setting is 4G, will bump it up
Thanks,
Dechang
> On Nov 19, 2015, at 10:50 AM, Ramana I N <in...@gmail.com> wrote:
>
> Can you make sure the configuration properties are all setup fine? Memory
> especially. I believe drill ships with a low default memory setting.
>
> Regards
> Ramana
>
>
> On Thu, Nov 19, 2015 at 10:18 AM, Abdel Hakim Deneche <adeneche@maprtech.com
>> wrote:
>
>> Having access to the query profiles would be really helpful.
>>
>> On Wed, Nov 18, 2015 at 11:53 AM, Dechang Gu <dg...@maprtech.com> wrote:
>>
>>> Hi All,
>>> With the help and support from Edmon, I am going to leverage the
>>> HPC resources at NICS ( https://www.nics.tennessee.edu/ <
>>> https://www.nics.tennessee.edu/>) for some Drill tests.
>>> The goal is to explore how Drill performs on HPC platform.
>>>
>>> 1. Platform configuration: details can be found here:
>>> https://www.nics.tennessee.edu/computing-resources/beacon/configuration
>> <
>>> https://www.nics.tennessee.edu/computing-resources/beacon/configuration>
>>>
>>> 2. Tests: TPCH and/or TPCDS (subset of queries)
>>> datasize: will start with SF1000 (~1TB), may increase
>> to
>>> larger dataset (depending on storage space available)
>>> test runs: 1 thread (like power run)
>>> multiple threads (like Throughput run)
>>> 3. Set up:
>>> — Drill (master) on Apache Hadoop
>>> — Drill (master) on MapR-FS
>>>
>>> Any comments/suggestions?
>>>
>>> Thanks,
>>> Dechang
>>
>>
>>
>>
>> --
>>
>> Abdelhakim Deneche
>>
>> Software Engineer
>>
>> <http://www.mapr.com/>
>>
>>
>> Now Available - Free Hadoop On-Demand Training
>> <
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>
>>
Re: Drill tests on HPC
Posted by Ramana I N <in...@gmail.com>.
Can you make sure the configuration properties are all setup fine? Memory
especially. I believe drill ships with a low default memory setting.
Regards
Ramana
On Thu, Nov 19, 2015 at 10:18 AM, Abdel Hakim Deneche <adeneche@maprtech.com
> wrote:
> Having access to the query profiles would be really helpful.
>
> On Wed, Nov 18, 2015 at 11:53 AM, Dechang Gu <dg...@maprtech.com> wrote:
>
> > Hi All,
> > With the help and support from Edmon, I am going to leverage the
> > HPC resources at NICS ( https://www.nics.tennessee.edu/ <
> > https://www.nics.tennessee.edu/>) for some Drill tests.
> > The goal is to explore how Drill performs on HPC platform.
> >
> > 1. Platform configuration: details can be found here:
> > https://www.nics.tennessee.edu/computing-resources/beacon/configuration
> <
> > https://www.nics.tennessee.edu/computing-resources/beacon/configuration>
> >
> > 2. Tests: TPCH and/or TPCDS (subset of queries)
> > datasize: will start with SF1000 (~1TB), may increase
> to
> > larger dataset (depending on storage space available)
> > test runs: 1 thread (like power run)
> > multiple threads (like Throughput run)
> > 3. Set up:
> > — Drill (master) on Apache Hadoop
> > — Drill (master) on MapR-FS
> >
> > Any comments/suggestions?
> >
> > Thanks,
> > Dechang
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: Drill tests on HPC
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
Having access to the query profiles would be really helpful.
On Wed, Nov 18, 2015 at 11:53 AM, Dechang Gu <dg...@maprtech.com> wrote:
> Hi All,
> With the help and support from Edmon, I am going to leverage the
> HPC resources at NICS ( https://www.nics.tennessee.edu/ <
> https://www.nics.tennessee.edu/>) for some Drill tests.
> The goal is to explore how Drill performs on HPC platform.
>
> 1. Platform configuration: details can be found here:
> https://www.nics.tennessee.edu/computing-resources/beacon/configuration <
> https://www.nics.tennessee.edu/computing-resources/beacon/configuration>
>
> 2. Tests: TPCH and/or TPCDS (subset of queries)
> datasize: will start with SF1000 (~1TB), may increase to
> larger dataset (depending on storage space available)
> test runs: 1 thread (like power run)
> multiple threads (like Throughput run)
> 3. Set up:
> — Drill (master) on Apache Hadoop
> — Drill (master) on MapR-FS
>
> Any comments/suggestions?
>
> Thanks,
> Dechang
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Drill tests on HPC
Posted by Parth Chandra <pa...@apache.org>.
@Steven, @Jacques, Any suggestions and/or thoughts on Dechang's proposal.
On Wed, Nov 18, 2015 at 11:53 AM, Dechang Gu <dg...@maprtech.com> wrote:
> Hi All,
> With the help and support from Edmon, I am going to leverage the
> HPC resources at NICS ( https://www.nics.tennessee.edu/ <
> https://www.nics.tennessee.edu/>) for some Drill tests.
> The goal is to explore how Drill performs on HPC platform.
>
> 1. Platform configuration: details can be found here:
> https://www.nics.tennessee.edu/computing-resources/beacon/configuration <
> https://www.nics.tennessee.edu/computing-resources/beacon/configuration>
>
> 2. Tests: TPCH and/or TPCDS (subset of queries)
> datasize: will start with SF1000 (~1TB), may increase to
> larger dataset (depending on storage space available)
> test runs: 1 thread (like power run)
> multiple threads (like Throughput run)
> 3. Set up:
> — Drill (master) on Apache Hadoop
> — Drill (master) on MapR-FS
>
> Any comments/suggestions?
>
> Thanks,
> Dechang