You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Xiaoyu Ma <hz...@corp.netease.com> on 2015/07/17 12:26:06 UTC
Hive Table with large number of partitions
Hi guys,
I saw when Hive Table object created it tries to load all existing partitions.
@transient val hiveQlPartitions: Seq[Partition] = table.getAllPartitions.map { p =>
val tPartition = new org.apache.hadoop.hive.metastore.api.Partition
tPartition.setDbName(databaseName)
tPartition.setTableName(tableName)
tPartition.setValues(p.values)
Above code for getAllPartitions costs minutes, tons of memory and failed almost always on table with large partitions. This made it useless in our use case.
I think this should be kind of lazy loading or can be postpone till partition pruning stage. Not sure why we need full partition info at this stage.
Any existing ticket to solve this?
马晓宇 / Xiaoyu Ma
hzmaxiaoyu@corp.netease.com
Re: Hive Table with large number of partitions
Posted by Michael Armbrust <mi...@databricks.com>.
https://github.com/apache/spark/pull/7421
On Fri, Jul 17, 2015 at 3:26 AM, Xiaoyu Ma <hz...@corp.netease.com>
wrote:
> Hi guys,
> I saw when Hive Table object created it tries to load all existing
> partitions.
>
>
> @transient val hiveQlPartitions: Seq[Partition] = table.getAllPartitions.map { p =>
> val tPartition = new org.apache.hadoop.hive.metastore.api.Partition
> tPartition.setDbName(databaseName)
> tPartition.setTableName(tableName)
> tPartition.setValues(p.values)
>
> Above code for getAllPartitions costs minutes, tons of memory and failed
> almost always on table with large partitions. This made it useless in our
> use case.
> I think this should be kind of lazy loading or can be postpone till
> partition pruning stage. Not sure why we need full partition info at this
> stage.
> Any existing ticket to solve this?
>
> 马晓宇 / Xiaoyu Ma
> hzmaxiaoyu@corp.netease.com
>
>
>
>
>