You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "jackyjfhu (via GitHub)" <gi...@apache.org> on 2025/02/27 04:04:30 UTC
[I] [Bug] when I use kyuubi's connector, Spark dynamic partitioning cannot be used. [kyuubi]
jackyjfhu opened a new issue, #6943:
URL: https://github.com/apache/kyuubi/issues/6943
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
kyuuiby:1.9.0
Spark：3.4.2
Currently, when connecting to hivemetastore and Spark through kyuubi-hive-connector and testing performance with TPC-DS, it is found that some SQL cannot be dynamically partitioned.
q33.sql
`with ss as (
select
i_manufact_id,sum(ss_ext_sales_price) total_sales
from
store_sales,
date_dim,
customer_address,
item
where
i_manufact_id in (select
i_manufact_id
from
item
where i_category in ('Home'))
and ss_item_sk = i_item_sk
and ss_sold_date_sk = d_date_sk
and d_year = 1998
and d_moy = 5
and ss_addr_sk = ca_address_sk
and ca_gmt_offset = -6
group by i_manufact_id),
cs as (
select
i_manufact_id,sum(cs_ext_sales_price) total_sales
from
catalog_sales,
date_dim,
customer_address,
item
where
i_manufact_id in (select
i_manufact_id
from
item
where i_category in ('Home'))
and cs_item_sk = i_item_sk
and cs_sold_date_sk = d_date_sk
and d_year = 1998
and d_moy = 5
and cs_bill_addr_sk = ca_address_sk
and ca_gmt_offset = -6
group by i_manufact_id),
ws as (
select
i_manufact_id,sum(ws_ext_sales_price) total_sales
from
web_sales,
date_dim,
customer_address,
item
where
i_manufact_id in (select
i_manufact_id
from
item
where i_category in ('Home'))
and ws_item_sk = i_item_sk
and ws_sold_date_sk = d_date_sk
and d_year = 1998
and d_moy = 5
and ws_bill_addr_sk = ca_address_sk
and ca_gmt_offset = -6
group by i_manufact_id)
select i_manufact_id ,sum(total_sales) total_sales
from (select * from ss
union all
select * from cs
union all
select * from ws) tmp1
group by i_manufact_id
order by total_sales
limit 100`
If kyuubi-hive-connecotor is not used, the execution plan will have dynamic partition pruning
<img width="802" alt="Image" src="https://github.com/user-attachments/assets/e2bca6a2-bcee-4b96-9e79-11642c29800c" />
If kyuubi-hive-connecotor is used，
there will be no dynamic partition pruning in the execution plan
<img width="925" alt="Image" src="https://github.com/user-attachments/assets/ceffe5b5-10de-4861-b5da-8f86cbb17305" />
### Affects Version(s)
1.9.0
### Kyuubi Server Log Output
```logtalk
```
### Kyuubi Engine Log Output
```logtalk
```
### Kyuubi Server Configurations
```yaml
```
### Kyuubi Engine Configurations
```yaml
spark.plugins com.datastrato.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.catalog.cos8lcu4e_hive.hadoop.hive.metastore.token.signature thrift://xxx:7004,thrift://xxxx:7004
spark.sql.catalog.cos8lcu4e_iceberg.hadoop.hive.metastore.token.signature thrift://xxxx:7004,thrift://xxxx:7004
spark.sql.catalog.default_catalog_hive_0o29qw56.hadoop.hive.metastore.token.signature thrift://xxx:40016
spark.sql.catalog.default_catalog_iceberg_0o29qw56.hadoop.hive.metastore.token.signature thrift://xxx:40016
spark.sql.catalog.kyuubi_cos8lcu4e_hive org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
spark.sql.catalog.kyuubi_cos8lcu4e_hive.hadoop.hive.metastore.token.signature thrift://xxx:7004,thrift://xxx:7004
spark.sql.catalog.kyuubi_cos8lcu4e_hive.hive.metastore.kerberos.principal hadoop/_HOST@TBDS-0O29QW56
spark.sql.catalog.kyuubi_cos8lcu4e_hive.hive.metastore.uris thrift://xxx:7004,thrift://xxxx:7004
spark.sql.catalog.kyuubi_default_catalog_hive_0o29qw56 org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
spark.sql.catalog.kyuubi_default_catalog_hive_0o29qw56.hadoop.hive.metastore.token.signature thrift://xxx:40016
spark.sql.catalog.kyuubi_default_catalog_hive_0o29qw56.hive.metastore.kerberos.principal hadoop/xxxx@xxx
spark.sql.catalog.kyuubi_default_catalog_hive_0o29qw56.hive.metastore.uris thrift://xxx:xxx
spark.sql.catalog.origin_cos8lcu4e_iceberg org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.origin_cos8lcu4e_iceberg.hadoop.hive.metastore.token.signature thrift://xxxx:7004,thrift://xxx:7004
spark.sql.catalog.origin_cos8lcu4e_iceberg.type hive
spark.sql.catalog.origin_cos8lcu4e_iceberg.uri thrift://xxx:7004,thrift://xxx:7004
spark.sql.catalog.origin_default_catalog_iceberg_0o29qw56 org.apache.iceberg.spark.SparkCatalog
```
### Additional context
Dynamic partition cutting core logic：
private def prune(plan: LogicalPlan): LogicalPlan = {
plan transformUp {
.......
var filterableScan = **getFilterableTableScan**(l, left)
if (filterableScan.isDefined && canPruneLeft(joinType) &&
hasPartitionPruningFilter(right)) {
newLeft = insertPredicate(l, newLeft, r, right, rightKeys, filterableScan.get)
} else {
filterableScan = getFilterableTableScan(r, right)
if (filterableScan.isDefined && canPruneRight(joinType) &&
hasPartitionPruningFilter(left) ) {
newRight = insertPredicate(r, newRight, l, left, leftKeys, filterableScan.get)
}
}
case _ =>
}
.......
Join(newLeft, newRight, joinType, Some(condition), hint)
}
}
def getFilterableTableScan(a: Expression, plan: LogicalPlan): Option[LogicalPlan] = {
val srcInfo: Option[(Expression, LogicalPlan)] = findExpressionAndTrackLineageDown(a, plan)
srcInfo.flatMap {
case (resExp, l: LogicalRelation) =>
l.relation match {
case fs: HadoopFsRelation =>
val partitionColumns = AttributeSet(
l.resolve(fs.partitionSchema, fs.sparkSession.sessionState.analyzer.resolver))
if (resExp.references.subsetOf(partitionColumns)) {
return Some(l)
} else {
None
}
case _ => None
}
case (resExp, l: HiveTableRelation) =>
if (resExp.references.subsetOf(AttributeSet(l.partitionCols))) {
return Some(l)
} else {
None
}
case (resExp, **r @ DataSourceV2ScanRelation(_, scan: SupportsRuntimeV2Filtering, _, _, _)**) =>
val filterAttrs = V2ExpressionUtils.resolveRefs[Attribute](scan.filterAttributes, r)
if (resExp.references.subsetOf(AttributeSet(filterAttrs))) {
Some(r)
} else {
None
}
case _ => None
}
}
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "maomaodev (via GitHub)" <gi...@apache.org>.
maomaodev commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-4393682142
@pan3793 Hi! After merging https://github.com/apache/kyuubi/pull/7130 and https://github.com/apache/kyuubi/pull/7123, for PARQUET/ORC Hive tables, KSHC will use `ParquetScan`/`ORCScan` from Spark datasource v2 by default. However, `ParquetScan`/`ORCScan` still do not support DPP. The same author, @flaming-archer, has opened PRs in the Spark community (see https://github.com/apache/spark/pull/52180 and https://github.com/apache/spark/pull/52009), but they have not been merged yet. I would like to ask your thoughts and suggestions:
1. Is it still necessary for KSHC `HiveScan` to support DPP?
2. For `ParquetScan`/`ORCScan`, should we keep pushing the fixes in the Spark community, or implement the functionality by wrapping `ParquetScan`/`ORCScan` inside KSHC?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "flaming-archer (via GitHub)" <gi...@apache.org>.
flaming-archer commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-5261966845
> Hello, can I ask if there is a planned Kyuubi version supporting this? (parquet, orc...)
>
> Currently, Apache Gravitino leverages Kyuubi to read Hive tables, but when Spark uses Hive catalog through Gravitino, it does not perform the DPP.
>
> The Gravitino community is [waiting for native support for this in Spark](https://github.com/apache/gravitino/issues/1578), but it seems spark community won't support DPP for DataSourceV2 in the near future...
I’m the author. We decided not to wait for the official merge and have already integrated this into our internal Spark version. Sharing it here as a reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] when I use kyuubi's connector, Spark dynamic partitioning cannot be used. [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-2686983072
The DPP requires the data source to implement either `SupportsRuntimeFiltering` or `SupportsRuntimeV2Filtering`, unfortunately, KSHC does not.
This is a feature request, not a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "pithecuse527 (via GitHub)" <gi...@apache.org>.
pithecuse527 commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-5253651233
Hello, can I ask if there is a planned Kyuubi version supporting this? (parquet, orc...)
Currently, Apache Gravitino leverages Kyuubi to read Hive tables, but when Spark uses Hive catalog through Gravitino, it does not perform the DPP.
The Gravitino community is [waiting for native support for this in Spark](https://github.com/apache/gravitino/issues/1578), but it seems spark community won't support DPP for DataSourceV2 in the near future...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "jackyjfhu (via GitHub)" <gi...@apache.org>.
jackyjfhu commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-2687035606
> The DPP requires the data source to implement either `SupportsRuntimeFiltering` or `SupportsRuntimeV2Filtering`, unfortunately, KSHC does not.
>
> This is a feature request, not a bug.
OK，thanks for reply
another question，when I execute this sql: ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS,I encountered the following error
<img width="1120" alt="Image" src="https://github.com/user-attachments/assets/9dc90772-edf0-4e6e-8b4c-5bd033a18a87" />
KSHC Through KSHC, Spark will parse the table into a V2 table
<img width="1245" alt="Image" src="https://github.com/user-attachments/assets/4f3e0f86-7d9e-4086-aa85-af591515f01d" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "ypcthenry (via GitHub)" <gi...@apache.org>.
ypcthenry commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-4310451341
[0001-KYUUBI-6943-Support-dynamic-partition-pruning-in-Spa.patch](https://github.com/user-attachments/files/27035767/0001-KYUUBI-6943-Support-dynamic-partition-pruning-in-Spa.patch)
I have implemented this feature code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "yangyuxia (via GitHub)" <gi...@apache.org>.
yangyuxia commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-2819952742
When we used tpcds orc 1T data to compare the performance of kyuubi and spark, we found that the execution time of kyuubi was 4 times that of spark. According to the physical execution plan, we found that kyuubi did not use the DPP and Predicate Pushdown optimization strategies. When will this function be implemented?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-4393770683
1. it's still valuable to make `HiveScan` support DPP, for non-parquet/orc tables.
2. it has little chance to get those chances into spark codebase shortly, we can do that in our side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "maomaodev (via GitHub)" <gi...@apache.org>.
maomaodev commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-4323827021
@ypcthenry Hello, could you submit a PR for this, or could we submit a PR together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #6943: Spark Hive connector supports dynamic partition prunning
URL: https://github.com/apache/kyuubi/issues/6943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] Spark Hive connector supports dynamic partition prunning [kyuubi]
Posted by "maomaodev (via GitHub)" <gi...@apache.org>.
maomaodev commented on issue #6943:
URL: https://github.com/apache/kyuubi/issues/6943#issuecomment-4393896248
> 1. it's still valuable to make `HiveScan` support DPP, for non-parquet/orc tables.
> 2. it has little chance to get those changes into spark codebase shortly, we can do that in our side
Ok, thanks. I will try to submit the PRs in two parts: first to implement `HiveScan`, and then to implement `ParquetScan`/`ORCScan`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org