You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Thulasi Ram Naidu Peddineni <th...@gmail.com> on 2011/10/20 22:10:18 UTC
Running hive on large number of files in S3
Hi All,
I have a use-case where I will be joining table1 with table2.
These are external tables with data in S3. table2 has many partitions
(say 10K) size being around 2GB and table1 has around 5-10 partitions
around 1-2MB. When I am joining these two tables, I observed that it
is taking lot of time to execute the query (more than 20 minutes).
>From my observation, the actual job execution is not taking lot of
time but the bottle neck is starting the job itself. This makes me
think that hive prefetching all the data from S3 and then do the
processing. Can some one explain me why is hive job is not starting
ontime on an external table with too many-partitions ?
One more observation here is, if I reduce the number of partitions
with same amount of data, the whole query is executing faster.
And what is the recommended way in such a scenario.
-----
Thanks,
Thulasi Ram P
Re: Running hive on large number of files in S3
Posted by Thulasi Ram Naidu Peddineni <th...@gmail.com>.
Thanks all for the inputs.
I will try using 0.7.1.
I will also revisit my partitioning logic and will try to reduce the
number of partitions.
-----
Thanks again,
Thulasi Ram P
On Fri, Oct 21, 2011 at 2:49 AM, Steven Wong <sw...@netflix.com> wrote:
> If you are using Amazon EMR, you can set hive.optimize.s3.query=true to
> speed up part b. See https://forums.aws.amazon.com/ann.jspa?annID=1105 for
> more info.
>
>
>
>
>
> From: Ashutosh Chauhan [mailto:hashutosh@apache.org]
> Sent: Thursday, October 20, 2011 1:21 PM
> To: user@hive.apache.org
> Subject: Re: Running hive on large number of files in S3
>
>
>
> Hey Thulasi,
>
>
>
> There are two factors which may affect job startup time in case of large
> number of partitions:
>
> a) Getting partition info from metastore: Hive stores metadata about each
> partiton in metastore. Depending on number of partitions, it needs to fetch,
> that can take some time.
>
> b) Input split calculation by Job client for all the map tasks. Majority of
> this will be spent in getting FileStatus for files from underlying
> filesystem.
>
>
>
> If your use case is that you process all the partitions then having fewer
> large partitions will help for both of these factors. On the other hand, if
> you process only few partitions usually, then doing finer
> grained partitioned is better.
>
>
>
> Hope it helps,
>
> Ashutosh
>
> On Thu, Oct 20, 2011 at 13:10, Thulasi Ram Naidu Peddineni
> <th...@gmail.com> wrote:
>
> Hi All,
> I have a use-case where I will be joining table1 with table2.
> These are external tables with data in S3. table2 has many partitions
> (say 10K) size being around 2GB and table1 has around 5-10 partitions
> around 1-2MB. When I am joining these two tables, I observed that it
> is taking lot of time to execute the query (more than 20 minutes).
> From my observation, the actual job execution is not taking lot of
> time but the bottle neck is starting the job itself. This makes me
> think that hive prefetching all the data from S3 and then do the
> processing. Can some one explain me why is hive job is not starting
> ontime on an external table with too many-partitions ?
> One more observation here is, if I reduce the number of partitions
> with same amount of data, the whole query is executing faster.
>
> And what is the recommended way in such a scenario.
>
> -----
> Thanks,
> Thulasi Ram P
>
>
RE: Running hive on large number of files in S3
Posted by Steven Wong <sw...@netflix.com>.
If you are using Amazon EMR, you can set hive.optimize.s3.query=true to speed up part b. See https://forums.aws.amazon.com/ann.jspa?annID=1105 for more info.
From: Ashutosh Chauhan [mailto:hashutosh@apache.org]
Sent: Thursday, October 20, 2011 1:21 PM
To: user@hive.apache.org
Subject: Re: Running hive on large number of files in S3
Hey Thulasi,
There are two factors which may affect job startup time in case of large number of partitions:
a) Getting partition info from metastore: Hive stores metadata about each partiton in metastore. Depending on number of partitions, it needs to fetch, that can take some time.
b) Input split calculation by Job client for all the map tasks. Majority of this will be spent in getting FileStatus for files from underlying filesystem.
If your use case is that you process all the partitions then having fewer large partitions will help for both of these factors. On the other hand, if you process only few partitions usually, then doing finer grained partitioned is better.
Hope it helps,
Ashutosh
On Thu, Oct 20, 2011 at 13:10, Thulasi Ram Naidu Peddineni <th...@gmail.com>> wrote:
Hi All,
I have a use-case where I will be joining table1 with table2.
These are external tables with data in S3. table2 has many partitions
(say 10K) size being around 2GB and table1 has around 5-10 partitions
around 1-2MB. When I am joining these two tables, I observed that it
is taking lot of time to execute the query (more than 20 minutes).
>From my observation, the actual job execution is not taking lot of
time but the bottle neck is starting the job itself. This makes me
think that hive prefetching all the data from S3 and then do the
processing. Can some one explain me why is hive job is not starting
ontime on an external table with too many-partitions ?
One more observation here is, if I reduce the number of partitions
with same amount of data, the whole query is executing faster.
And what is the recommended way in such a scenario.
-----
Thanks,
Thulasi Ram P
Re: Running hive on large number of files in S3
Posted by Ashutosh Chauhan <ha...@apache.org>.
Hey Thulasi,
There are two factors which may affect job startup time in case of large
number of partitions:
a) Getting partition info from metastore: Hive stores metadata about each
partiton in metastore. Depending on number of partitions, it needs to fetch,
that can take some time.
b) Input split calculation by Job client for all the map tasks. Majority of
this will be spent in getting FileStatus for files from underlying
filesystem.
If your use case is that you process all the partitions then having fewer
large partitions will help for both of these factors. On the other hand, if
you process only few partitions usually, then doing finer
grained partitioned is better.
Hope it helps,
Ashutosh
On Thu, Oct 20, 2011 at 13:10, Thulasi Ram Naidu Peddineni <
thulasiram333@gmail.com> wrote:
> Hi All,
> I have a use-case where I will be joining table1 with table2.
> These are external tables with data in S3. table2 has many partitions
> (say 10K) size being around 2GB and table1 has around 5-10 partitions
> around 1-2MB. When I am joining these two tables, I observed that it
> is taking lot of time to execute the query (more than 20 minutes).
> From my observation, the actual job execution is not taking lot of
> time but the bottle neck is starting the job itself. This makes me
> think that hive prefetching all the data from S3 and then do the
> processing. Can some one explain me why is hive job is not starting
> ontime on an external table with too many-partitions ?
> One more observation here is, if I reduce the number of partitions
> with same amount of data, the whole query is executing faster.
>
> And what is the recommended way in such a scenario.
>
> -----
> Thanks,
> Thulasi Ram P
>
Re: Running hive on large number of files in S3
Posted by Jerome Boulon <jb...@netflix.com>.
Hi,
I don't think that your job is actually prefetching the data while you're
waiting.
If you have a large number of partitions then getting the list of files to
compute the split
(aka prefetching the filenames from S3) is what is taking for ever.
If you have a premium support from amazon you may want to ask for help in
this area.
/Jerome
On 10/20/11 1:10 PM, "Thulasi Ram Naidu Peddineni"
<th...@gmail.com> wrote:
>Hi All,
> I have a use-case where I will be joining table1 with table2.
>These are external tables with data in S3. table2 has many partitions
>(say 10K) size being around 2GB and table1 has around 5-10 partitions
>around 1-2MB. When I am joining these two tables, I observed that it
>is taking lot of time to execute the query (more than 20 minutes).
>From my observation, the actual job execution is not taking lot of
>time but the bottle neck is starting the job itself. This makes me
>think that hive prefetching all the data from S3 and then do the
>processing. Can some one explain me why is hive job is not starting
>ontime on an external table with too many-partitions ?
> One more observation here is, if I reduce the number of partitions
>with same amount of data, the whole query is executing faster.
>
>And what is the recommended way in such a scenario.
>
>-----
>Thanks,
>Thulasi Ram P
>