You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Xiaoyu Ma <hz...@corp.netease.com> on 2015/07/17 12:26:06 UTC

Hive Table with large number of partitions

Hi guys,
I saw when Hive Table object created it tries to load all existing partitions. 

@transient val hiveQlPartitions: Seq[Partition] = table.getAllPartitions.map { p =>
  val tPartition = new org.apache.hadoop.hive.metastore.api.Partition
  tPartition.setDbName(databaseName)
  tPartition.setTableName(tableName)
  tPartition.setValues(p.values)
Above code for getAllPartitions costs minutes, tons of memory and failed almost always on table with large partitions. This made it useless in our use case. 
I think this should be kind of lazy loading or can be postpone till partition pruning stage. Not sure why we need full partition info at this stage.
Any existing ticket to solve this? 

马晓宇 / Xiaoyu Ma
hzmaxiaoyu@corp.netease.com





Re: Hive Table with large number of partitions

Posted by Michael Armbrust <mi...@databricks.com>.
https://github.com/apache/spark/pull/7421

On Fri, Jul 17, 2015 at 3:26 AM, Xiaoyu Ma <hz...@corp.netease.com>
wrote:

> Hi guys,
> I saw when Hive Table object created it tries to load all existing
> partitions.
>
>
> @transient val hiveQlPartitions: Seq[Partition] = table.getAllPartitions.map { p =>
>   val tPartition = new org.apache.hadoop.hive.metastore.api.Partition
>   tPartition.setDbName(databaseName)
>   tPartition.setTableName(tableName)
>   tPartition.setValues(p.values)
>
> Above code for getAllPartitions costs minutes, tons of memory and failed
> almost always on table with large partitions. This made it useless in our
> use case.
> I think this should be kind of lazy loading or can be postpone till
> partition pruning stage. Not sure why we need full partition info at this
> stage.
> Any existing ticket to solve this?
>
> 马晓宇 / Xiaoyu Ma
> hzmaxiaoyu@corp.netease.com
>
>
>
>
>