You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "Kwafoor (via GitHub)" <gi...@apache.org> on 2023/12/01 12:49:17 UTC
[PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Kwafoor opened a new pull request, #5800:
URL: https://github.com/apache/kyuubi/pull/5800
# :mag: Description
## Issue References ๐
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
This pull request fixes #5467
## Describe Your Solution ๐ง
1. Add Gluten UTs.
2. Setup CI for Gluten testing
3. Write docs to guide users in setting up Kyuubi with Spark plus Gluten.
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklists
## ๐ Author Self Checklist
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## ๐ Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "Kwafoor (via GitHub)" <gi...@apache.org>.
Kwafoor commented on code in PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#discussion_r1413474846
##########
extensions/spark/kyuubi-spark-connector-tpcds/pom.xml:
##########
@@ -213,4 +213,29 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+
+ <profiles>
+ <profile>
+ <id>gluten</id>
Review Comment:
good idea, i will implement this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#issuecomment-1844127956
thanks, merged to master. We shoule keep check nightly build to see if the workflow work well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "Kwafoor (via GitHub)" <gi...@apache.org>.
Kwafoor commented on PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#issuecomment-1837757189
Is there somthing wrong with this unit tests? Is it necessary to open a new issue to fix?(seems to happen frequently.)
```
EngineRefWithEtcdSuite:
- CONNECTION shared level engine name
- USER shared level engine name
- GROUP shared level engine name
- SERVER shared level engine name
- check the engine space of engine pool
- start and get engine address with lock
Exception in thread "pool-197-thread-1" java.lang.NullPointerException
at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:237)
at org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient.tryWithLock(EtcdDiscoveryClient.scala:187)
at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:174)
at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:179)
at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:306)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:306)
at org.apache.kyuubi.engine.EngineRefTests.$anonfun$$init$$17(EngineRefTests.scala:298)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.engine.EngineRefTests.$anonfun$$init$$16(EngineRefTests.scala:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Error: Exception in thread "pool-197-thread-2" java.lang.Error: org.apache.kyuubi.KyuubiSQLException: Timeout to lock on path [/engine_test2_1.9.0-SNAPSHOT_USER_SPARK_SQL_lock/runner/default] after 4500 ms. There would be some problem that other session may create engine timeout.
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1155)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.kyuubi.KyuubiSQLException: Timeout to lock on path [/engine_test2_1.9.0-SNAPSHOT_USER_SPARK_SQL_lock/runner/default] after 4500 ms. There would be some problem that other session may create engine timeout.
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:70)
at org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient.tryWithLock(EtcdDiscoveryClient.scala:181)
at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:174)
at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:179)
at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:306)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:306)
at org.apache.kyuubi.engine.EngineRefTests.$anonfun$$init$$17(EngineRefTests.scala:298)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.engine.EngineRefTests.$anonfun$$init$$16(EngineRefTests.scala:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
... 2 more
Exception in thread "pool-197-thread-3" java.lang.NullPointerException
at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:237)
at org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient.tryWithLock(EtcdDiscoveryClient.scala:187)
at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:174)
at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:179)
at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:306)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:306)
at org.apache.kyuubi.engine.EngineRefTests.$anonfun$$init$$17(EngineRefTests.scala:298)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.engine.EngineRefTests.$anonfun$$init$$16(EngineRefTests.scala:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#discussion_r1413430251
##########
pom.xml:
##########
@@ -236,7 +236,7 @@
<maven.plugin.frontend.version>1.12.1</maven.plugin.frontend.version>
<maven.plugin.scala.version>4.8.0</maven.plugin.scala.version>
<maven.plugin.scalatest.version>2.2.0</maven.plugin.scalatest.version>
- <maven.plugin.scalatest.exclude.tags>org.scalatest.tags.Slow,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
+ <maven.plugin.scalatest.exclude.tags>org.scalatest.tags.Slow,org.apache.kyuubi.tags.GlutenTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
Review Comment:
So we do not need these changes after we add a new module. By default, we do not include that new module.
##########
extensions/spark/kyuubi-spark-connector-tpcds/pom.xml:
##########
@@ -213,4 +213,29 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+
+ <profiles>
+ <profile>
+ <id>gluten</id>
Review Comment:
Can we make a new module for Gluten IT ? Includes tpcds and tpch related change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "Kwafoor (via GitHub)" <gi...@apache.org>.
Kwafoor commented on PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#issuecomment-1842416914
latest gluten ci test:https://github.com/Kwafoor/incubator-kyuubi/actions/runs/7111586978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you closed pull request #5800: [KYUUBI#5467] Integrate Intel Gluten with Spark engine
URL: https://github.com/apache/kyuubi/pull/5800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#issuecomment-1837807246
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5800?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`8f529aa`)](https://app.codecov.io/gh/apache/kyuubi/commit/8f529aacf1d8cd1d6e68f5692ad3532b71b2565b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.41% compared to head [(`1de45d7`)](https://app.codecov.io/gh/apache/kyuubi/pull/5800?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.34%.
> Report is 3 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #5800 +/- ##
============================================
- Coverage 61.41% 61.34% -0.08%
Complexity 23 23
============================================
Files 607 608 +1
Lines 35948 35931 -17
Branches 4939 4937 -2
============================================
- Hits 22078 22041 -37
- Misses 11484 11497 +13
- Partials 2386 2393 +7
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/5800?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#issuecomment-1844189950
late LGTM. thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#discussion_r1417191052
##########
integration-tests/kyuubi-gluten-it/pom.xml:
##########
@@ -0,0 +1,126 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-parent</artifactId>
+ <version>1.9.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
Review Comment:
`../pom.xml`
##########
integration-tests/kyuubi-gluten-it/src/test/resources/log4j.properties:
##########
Review Comment:
Why do you need log4j properties as we have log4j2 xml ?
##########
integration-tests/kyuubi-gluten-it/pom.xml:
##########
@@ -0,0 +1,126 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-parent</artifactId>
+ <version>1.9.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+
+ <artifactId>kyuubi-gluten-it_${scala.binary.version}</artifactId>
+ <name>Kyuubi Test Gluten IT</name>
+ <url>https://kyuubi.apache.org/</url>
+
+ <properties>
+ <spark.version>3.4.1</spark.version>
+ <spark.binary.version>3.4</spark.binary.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-common_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-common_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-spark-connector-common_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-spark-connector-tpcds_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-spark-connector-tpch_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-hive_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
+ <testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
+ </build>
+
+ <profiles>
+ <profile>
+ <id>gluten-spark-3.4</id>
+ <properties>
+ <maven.plugin.scalatest.include.tags>org.apache.kyuubi.tags.GlutenTest</maven.plugin.scalatest.include.tags>
+ <spark.version>3.4.1</spark.version>
+ <spark.binary.version>3.4</spark.binary.version>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>io.glutenproject</groupId>
+ <artifactId>gluten-velox-bundle-spark3.4_2.12-ubuntu_22.04</artifactId>
+ <version>1.1.0-SNAPSHOT</version>
Review Comment:
shall we pull out gluten version property ? to make easy change in future
##########
docs/deployment/spark/gluten.md:
##########
Review Comment:
shall we add it to `index.rst` ?
##########
integration-tests/kyuubi-gluten-it/src/test/scala/org/apache/kyuubi/it/gluten/tpcds/GlutenTPCDSQuerySuite.scala:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.it.gluten.tpcds
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.SparkSession
+import org.scalatest.tags.Slow
+
+import org.apache.kyuubi.{GlutenSuiteMixin, KyuubiFunSuite}
+import org.apache.kyuubi.it.gluten.TPCUtils.loadTPCFile
+import org.apache.kyuubi.spark.connector.common.GoldenFileUtils.LICENSE_HEADER
+import org.apache.kyuubi.spark.connector.common.LocalSparkSession.withSparkSession
+import org.apache.kyuubi.spark.connector.tpcds.TPCDSCatalog
+import org.apache.kyuubi.tags.GlutenTest
+
+@Slow
+@GlutenTest
+class GlutenTPCDSQuerySuite extends KyuubiFunSuite with GlutenSuiteMixin {
+
+ val queries: Set[String] = (1 to 99).map(i => s"q$i").toSet -
+ ("q14", "q23", "q24", "q39") +
+ ("q14a", "q14b", "q23a", "q23b", "q24a", "q24b", "q39a", "q39b") -
+ // TODO:Fix gluten tpc-ds query test
+ ("q1", "q4", "q7", "q11", "q12", "q17", "q20", "q21", "q25", "q26", "q29", "q30", "q34", "q37",
+ "q39a", "q39b", "q40", "q43", "q46", "q49", "q56", "q58", "q59", "q60", "q68", "q73", "q74",
+ "q78", "q79", "q81", "q82", "q83", "q84", "q91", "q98")
Review Comment:
surprised, so many tests can not pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "Kwafoor (via GitHub)" <gi...@apache.org>.
Kwafoor commented on code in PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#discussion_r1413314096
##########
kyuubi-common/src/test/scala/org/apache/kyuubi/GlutenSuiteMixin.scala:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi
+
+trait GlutenSuiteMixin {
+ protected def extraJars: String = {
+ System.getProperty("java.class.path")
+ .split(":")
+ .filter(_.contains("gluten-velox-bundle-spark")).head
+ }
+
+ protected def extraConfigs = Map(
+ "spark.plugins" -> "io.glutenproject.GlutenPlugin",
+ "spark.memory.offHeap.size" -> "20g",
Review Comment:
copied this from a gluten demo, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI#5467] Integrate Intel Gluten with Spark engine [kyuubi]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #5800:
URL: https://github.com/apache/kyuubi/pull/5800#discussion_r1412709956
##########
kyuubi-common/src/test/scala/org/apache/kyuubi/GlutenSuiteMixin.scala:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi
+
+trait GlutenSuiteMixin {
+ protected def extraJars: String = {
+ System.getProperty("java.class.path")
+ .split(":")
+ .filter(_.contains("gluten-velox-bundle-spark")).head
+ }
+
+ protected def extraConfigs = Map(
+ "spark.plugins" -> "io.glutenproject.GlutenPlugin",
+ "spark.memory.offHeap.size" -> "20g",
Review Comment:
the GitHub Action has a memory limit of up to 7GB. see [docs](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org