You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by mb...@datameer.com on 2019/11/08 15:44:18 UTC
Gather Partition Locations
Hi,
I have a question about how to get the location for a bunch of partitions.
My answer is: using the hive query `DESCRIBE EXTENDED <tablename> PARTITION(<partition_name>)`
I'm getting back a json response (if i set return type to JSON) which has the
HDFS location in it.
But. If I have, lets say 1000 partitions and every query needs 0.5 sec,
I have to wait 500 sec.
So my question is, do you have a single query to gather all locations?
Or do you have a workaround to get the locations faster?
I think about to query the metadatore RDS directly. Similar to
http://www.openkb.info/2015/04/how-to-list-table-or-partition-location.html
But in an enterprise environment I'm pretty sure this approach would not be
the best because the RDS (mysql or derby) is maybe not reachable or
I don't have the permission to it.
Any other hint or idea how to get all partition locations from lets
say external table with custom partition locations in a performant way.
Thanks
Marko
Re: Gather Partition Locations
Posted by mb...@datameer.com.
Vivek Shrivastava <vi...@gmail.com> wrote:
> If you have access to HCatalog, it also has jdbc connection that would
> allow you to get faster response.
ah ok. sounds awesome as well! I will check.
thanks
marko
--
Marko Bauhardt
Software Engineer
www.datameer.com
Phone: +49 345 279 5030
Datameer GmbH
Magdeburger Str. 23, 06112 Halle (Saale), Germany
Amtsgericht Stendal, HRB: 10348, Geschäftsführer: Frank Henze
Re: Gather Partition Locations
Posted by Vivek Shrivastava <vi...@gmail.com>.
If you have access to HCatalog, it also has jdbc connection that would
allow you to get faster response.
On Tue, Nov 12, 2019 at 6:53 AM Elliot West <te...@gmail.com> wrote:
> Hello,
>
> We faced a similar problem. Additionally, we had job clients were
> difficult to integrate directly with the Thirft API, but needed to resolve
> file locations via the metastore. To handle this, we build a cut down
> service with a REST API that fronts the Hive metastore. The API is
> optimised for this specific case of retrieving lists of file locations for
> some set of partitions. A nice thing about this approach is that our users
> are able to fetch file lists that are easy to parse, using very simple
> integrations (a curl command for example).
>
> Cheers,
>
> Elliot.
>
> On Fri, 8 Nov 2019 at 15:44, <mb...@datameer.com> wrote:
>
>> Hi,
>> I have a question about how to get the location for a bunch of partitions.
>> My answer is: using the hive query `DESCRIBE EXTENDED <tablename>
>> PARTITION(<partition_name>)`
>>
>> I'm getting back a json response (if i set return type to JSON) which has
>> the
>> HDFS location in it.
>>
>> But. If I have, lets say 1000 partitions and every query needs 0.5 sec,
>> I have to wait 500 sec.
>>
>> So my question is, do you have a single query to gather all locations?
>> Or do you have a workaround to get the locations faster?
>> I think about to query the metadatore RDS directly. Similar to
>>
>> http://www.openkb.info/2015/04/how-to-list-table-or-partition-location.html
>> But in an enterprise environment I'm pretty sure this approach would not
>> be
>> the best because the RDS (mysql or derby) is maybe not reachable or
>> I don't have the permission to it.
>>
>> Any other hint or idea how to get all partition locations from lets
>> say external table with custom partition locations in a performant way.
>>
>> Thanks
>> Marko
>>
>>
>>
Re: Gather Partition Locations
Posted by Elliot West <te...@gmail.com>.
Hello,
We faced a similar problem. Additionally, we had job clients were difficult
to integrate directly with the Thirft API, but needed to resolve file
locations via the metastore. To handle this, we build a cut down service
with a REST API that fronts the Hive metastore. The API is optimised for
this specific case of retrieving lists of file locations for some set of
partitions. A nice thing about this approach is that our users are able to
fetch file lists that are easy to parse, using very simple integrations (a
curl command for example).
Cheers,
Elliot.
On Fri, 8 Nov 2019 at 15:44, <mb...@datameer.com> wrote:
> Hi,
> I have a question about how to get the location for a bunch of partitions.
> My answer is: using the hive query `DESCRIBE EXTENDED <tablename>
> PARTITION(<partition_name>)`
>
> I'm getting back a json response (if i set return type to JSON) which has
> the
> HDFS location in it.
>
> But. If I have, lets say 1000 partitions and every query needs 0.5 sec,
> I have to wait 500 sec.
>
> So my question is, do you have a single query to gather all locations?
> Or do you have a workaround to get the locations faster?
> I think about to query the metadatore RDS directly. Similar to
> http://www.openkb.info/2015/04/how-to-list-table-or-partition-location.html
> But in an enterprise environment I'm pretty sure this approach would not be
> the best because the RDS (mysql or derby) is maybe not reachable or
> I don't have the permission to it.
>
> Any other hint or idea how to get all partition locations from lets
> say external table with custom partition locations in a performant way.
>
> Thanks
> Marko
>
>
>
Re: Gather Partition Locations
Posted by mb...@datameer.com.
Ashutosh Bapat <as...@gmail.com> wrote:
> There are multiple ways
> 1. Query the HiveMetaStore directly.
do you mean via thrift client? or directly native jdbc?
But i think this is in an enterprise env not possible, when i'm not on the same machine where hive server is running.
i believe the mysql or postgres service will not be exposed.
> 2. Use sys.* tables or better even information_schema to get the same.
ok. as far as I understand this is available since hive 3 or as part of hive (e.g. beeline)?
https://issues.apache.org/jira/browse/HIVE-1010
so no chance to do it in hive < 3 or?
thanks for your hints, very helpful.
marko
Re: Gather Partition Locations
Posted by Ashutosh Bapat <as...@gmail.com>.
There are multiple ways
1. Query the HiveMetaStore directly.
2. Use sys.* tables or better even information_schema to get the same.
On Fri, Nov 8, 2019 at 9:14 PM <mb...@datameer.com> wrote:
> Hi,
> I have a question about how to get the location for a bunch of partitions.
> My answer is: using the hive query `DESCRIBE EXTENDED <tablename>
> PARTITION(<partition_name>)`
>
> I'm getting back a json response (if i set return type to JSON) which has
> the
> HDFS location in it.
>
> But. If I have, lets say 1000 partitions and every query needs 0.5 sec,
> I have to wait 500 sec.
>
> So my question is, do you have a single query to gather all locations?
> Or do you have a workaround to get the locations faster?
> I think about to query the metadatore RDS directly. Similar to
> http://www.openkb.info/2015/04/how-to-list-table-or-partition-location.html
> But in an enterprise environment I'm pretty sure this approach would not be
> the best because the RDS (mysql or derby) is maybe not reachable or
> I don't have the permission to it.
>
> Any other hint or idea how to get all partition locations from lets
> say external table with custom partition locations in a performant way.
>
> Thanks
> Marko
>
>
>
--
--
Best Wishes,
Ashutosh Bapat