You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "gabrywu (via GitHub)" <gi...@apache.org> on 2024/09/13 09:03:08 UTC
[PR] Compact table [kyuubi]
gabrywu opened a new pull request, #6695:
URL: https://github.com/apache/kyuubi/pull/6695
# :mag: Description
## Issue References 🔗
This pull request closing #6691
## Describe Your Solution 🔧
There are many cases in which a SQL generate small files, we MUST merge them into bigger ones.
I create a new Spark SQL command to merge small files, which doesn't read-write all of the records of a table, it just merges files in a binary level. Take a CSV table for example, it only appends the byte array from one file to another one, without reading & writing records
Syntax here
```sparksql
compact table table_name [INTO ${targetFileSize} ${targetFileSizeUnit} ] [ cleanup | retain | list ]
-- targetFileSizeUnit can be 'b','k','m','g','t','p'
-- cleanup means cleaning compact staging folders, which contains original small files, default behavior
-- retain means retaining compact staging folders, for testing, and we can recover with the staging data
-- list means this command only get the merging result, and don't run actually
```
```sparksql
recover compact table table_name
-- recover a table if compact table command fails
```
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist 📝
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on code in PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#discussion_r1759725468
##########
extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/compact/CachePerformanceViewCommand.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.sql.compact
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.spark.sql.{Row, SparkInternalExplorer, SparkSession}
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.command.{DropTableCommand, LeafRunnableCommand}
+
+case class CachePerformanceViewCommand(
+ tableIdentifier: Seq[String],
+ performancePlan: LogicalPlan,
+ originalFileLocations: Seq[String],
+ options: CompactTableOption) extends LeafRunnableCommand {
+
+ override def innerChildren: Seq[QueryPlan[_]] = Seq(performancePlan)
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val dropViewCommand = DropTableCommand(
+ CompactTableUtils.getTableIdentifier(tableIdentifier),
+ ifExists = true,
+ isView = true,
+ purge = true)
+ dropViewCommand.run(sparkSession)
+
+ val speculation =
+ sparkSession.sparkContext.getConf.getBoolean("spark.speculation", defaultValue = false)
+ if (speculation) {
+ sparkSession.sparkContext.getConf.set("spark.speculation", "false")
Review Comment:
I think it can, if the upcoming TaskScheduler uses the same SparkContext instance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2370405113
do you support to compact one partition for partitioned table？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] A new Spark SQL command to merge small files [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2351431586
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/6695?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `582 lines` in your changes missing coverage. Please review.
> Project coverage is 0.00%. Comparing base [(`353877b`)](https://app.codecov.io/gh/apache/kyuubi/commit/353877b328e35902e59a7b0a11ea99592236c2d3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`257dfd6`)](https://app.codecov.io/gh/apache/kyuubi/commit/257dfd692c66bb5a796b3c22d5ffe12d5492f7ae?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 19 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/kyuubi/pull/6695?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/kyuubi/sql/compact/SmallFileCollectExec.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsql%2Fcompact%2FSmallFileCollectExec.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zcWwvY29tcGFjdC9TbWFsbEZpbGVDb2xsZWN0RXhlYy5zY2FsYQ==) | 0.00% | [75 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../kyuubi/sql/compact/merge/AbstractFileMerger.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsql%2Fcompact%2Fmerge%2FAbstractFileMerger.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zcWwvY29tcGFjdC9tZXJnZS9BYnN0cmFjdEZpbGVNZXJnZXIuc2NhbGE=) | 0.00% | [58 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...yuubi/sql/compact/RecoverCompactTableCommand.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsql%2Fcompact%2FRecoverCompactTableCommand.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zcWwvY29tcGFjdC9SZWNvdmVyQ29tcGFjdFRhYmxlQ29tbWFuZC5zY2FsYQ==) | 0.00% | [46 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/kyuubi/sql/compact/CompactTableResolver.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsql%2Fcompact%2FCompactTableResolver.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zcWwvY29tcGFjdC9Db21wYWN0VGFibGVSZXNvbHZlci5zY2FsYQ==) | 0.00% | [43 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/org/apache/kyuubi/sql/compact/CompactTable.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsql%2Fcompact%2FCompactTable.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zcWwvY29tcGFjdC9Db21wYWN0VGFibGUuc2NhbGE=) | 0.00% | [38 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/kyuubi/sql/compact/merge/ParquetFileMerger.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsql%2Fcompact%2Fmerge%2FParquetFileMerger.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zcWwvY29tcGFjdC9tZXJnZS9QYXJxdWV0RmlsZU1lcmdlci5zY2FsYQ==) | 0.00% | [38 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...apache/kyuubi/sql/compact/SmallFileMergeExec.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsql%2Fcompact%2FSmallFileMergeExec.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zcWwvY29tcGFjdC9TbWFsbEZpbGVNZXJnZUV4ZWMuc2NhbGE=) | 0.00% | [34 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...uubi/sql/compact/CachePerformanceViewCommand.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsql%2Fcompact%2FCachePerformanceViewCommand.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zcWwvY29tcGFjdC9DYWNoZVBlcmZvcm1hbmNlVmlld0NvbW1hbmQuc2NhbGE=) | 0.00% | [33 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/org/apache/spark/sql/SparkInternalExplorer.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2FSparkInternalExplorer.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL3NwYXJrL3NxbC9TcGFya0ludGVybmFsRXhwbG9yZXIuc2NhbGE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../apache/kyuubi/sql/compact/CompactTableUtils.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&filepath=extensions%2Fspark%2Fkyuubi-extension-spark-3-5%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsql%2Fcompact%2FCompactTableUtils.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1YmktZXh0ZW5zaW9uLXNwYXJrLTMtNS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zcWwvY29tcGFjdC9Db21wYWN0VGFibGVVdGlscy5zY2FsYQ==) | 0.00% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [12 more](https://app.codecov.io/gh/apache/kyuubi/pull/6695?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #6695 +/- ##
=======================================
Coverage 0.00% 0.00%
=======================================
Files 682 706 +24
Lines 42192 42861 +669
Branches 5755 5847 +92
=======================================
- Misses 42192 42861 +669
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/6695?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#discussion_r1764576293
##########
extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/compact/merge/PlainFileLikeMerger.scala:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.sql.compact.merge
+
+import scala.util.Try
+
+import org.apache.hadoop.fs.{FileSystem, Path => HadoopPath}
+import org.apache.hadoop.io.IOUtils
+
+import org.apache.kyuubi.sql.compact.MergingFile
+
+class PlainFileLikeMerger(dataSource: String, codec: Option[String])
+ extends AbstractFileMerger(dataSource, codec, true) {
+ override protected def mergeFiles(
+ fileSystem: FileSystem,
+ smallFiles: List[MergingFile],
+ mergedFileInStaging: HadoopPath): Try[HadoopPath] = Try {
+ val smallFilePaths = smallFiles.map(r => new HadoopPath(location, r.name))
+ val fos = fileSystem.create(mergedFileInStaging, false)
+ try {
+ smallFilePaths.foreach { f =>
+ val is = fileSystem.open(f)
+ IOUtils.copyBytes(is, fos, hadoopConf, false)
+ IOUtils.closeStream(is)
Review Comment:
finnlay closeStream
##########
extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/compact/merge/PlainFileLikeMerger.scala:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.sql.compact.merge
+
+import scala.util.Try
+
+import org.apache.hadoop.fs.{FileSystem, Path => HadoopPath}
+import org.apache.hadoop.io.IOUtils
+
+import org.apache.kyuubi.sql.compact.MergingFile
+
+class PlainFileLikeMerger(dataSource: String, codec: Option[String])
+ extends AbstractFileMerger(dataSource, codec, true) {
+ override protected def mergeFiles(
+ fileSystem: FileSystem,
+ smallFiles: List[MergingFile],
+ mergedFileInStaging: HadoopPath): Try[HadoopPath] = Try {
+ val smallFilePaths = smallFiles.map(r => new HadoopPath(location, r.name))
+ val fos = fileSystem.create(mergedFileInStaging, false)
+ try {
+ smallFilePaths.foreach { f =>
+ val is = fileSystem.open(f)
+ IOUtils.copyBytes(is, fos, hadoopConf, false)
+ IOUtils.closeStream(is)
Review Comment:
finally closeStream
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2360691899
An email thread to decide which one should be used, command or call procedure
https://lists.apache.org/list?dev@kyuubi.apache.org:lte=1M:[VOTE][DISCUSS]%20A%20Spark%20SQL%20command%20or%20Call%20procedure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2373524363
> do you support to compact one partition for partitioned table？
hi, @turboFei I removed this feature from this PR. only support partition table internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#discussion_r1764602436
##########
extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/compact/CachePerformanceViewCommand.scala:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.sql.compact
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.spark.sql.{Row, SparkInternalExplorer, SparkSession}
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.command.{DropTableCommand, LeafRunnableCommand}
+
+case class CachePerformanceViewCommand(
+ tableIdentifier: Seq[String],
+ performancePlan: LogicalPlan,
+ originalFileLocations: Seq[String],
+ options: CompactTableOption) extends LeafRunnableCommand {
+
+ override def innerChildren: Seq[QueryPlan[_]] = Seq(performancePlan)
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val dropViewCommand = DropTableCommand(
+ CompactTableUtils.getTableIdentifier(tableIdentifier),
+ ifExists = true,
+ isView = true,
+ purge = true)
+ dropViewCommand.run(sparkSession)
+
+ val speculation =
+ sparkSession.sparkContext.getConf.getBoolean(
+ SparkInternalExplorer.SPECULATION_ENABLED_SYNONYM.key,
+ defaultValue = false)
+ if (speculation) {
+ sparkSession.sparkContext.getConf.set(
+ SparkInternalExplorer.SPECULATION_ENABLED_SYNONYM.key,
+ "false")
+ log.warn("set spark.speculation to false")
+ }
+
+ val cacheTableCommand =
+ SparkInternalExplorer.CacheTableAsSelectExec(tableIdentifier.head, performancePlan)
+
+ // this result always empty
+ cacheTableCommand.run()
+
+ if (options == CompactTableOptions.CleanupStagingFolder) {
+ val fileSystem = FileSystem.get(sparkSession.sparkContext.hadoopConfiguration)
+ originalFileLocations.foreach { originalFileLocation =>
+ val compactStagingDir = CompactTableUtils.getCompactStagingDir(originalFileLocation)
+ fileSystem.delete(compactStagingDir, true)
+ }
+
+ }
+ if (speculation) {
+ sparkSession.sparkContext.getConf.set(
Review Comment:
Do we need to rollback regardless of the success or failure of the command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2375540727
Close this PR and will create a new one if apache/spark/pull/47190 commit is released in next Spark version v4.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#discussion_r1764597278
##########
extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/compact/CompressionCodecsUtil.scala:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.sql.compact
+
+object CompressionCodecsUtil {
Review Comment:
not used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2352071503
| Table Format | Codec | Supported |
|--------------|------------|-----------|
| parquet | all codecs | Y |
| avro | snappy | Y |
| json | gzip | Y |
| csv | gzip | Y |
| text | gzip | Y |
| orc | snappy | Y |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on code in PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#discussion_r1759725468
##########
extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/compact/CachePerformanceViewCommand.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.sql.compact
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.spark.sql.{Row, SparkInternalExplorer, SparkSession}
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.command.{DropTableCommand, LeafRunnableCommand}
+
+case class CachePerformanceViewCommand(
+ tableIdentifier: Seq[String],
+ performancePlan: LogicalPlan,
+ originalFileLocations: Seq[String],
+ options: CompactTableOption) extends LeafRunnableCommand {
+
+ override def innerChildren: Seq[QueryPlan[_]] = Seq(performancePlan)
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val dropViewCommand = DropTableCommand(
+ CompactTableUtils.getTableIdentifier(tableIdentifier),
+ ifExists = true,
+ isView = true,
+ purge = true)
+ dropViewCommand.run(sparkSession)
+
+ val speculation =
+ sparkSession.sparkContext.getConf.getBoolean("spark.speculation", defaultValue = false)
+ if (speculation) {
+ sparkSession.sparkContext.getConf.set("spark.speculation", "false")
Review Comment:
I think it can, if the upcoming TaskSetManager uses the same SparkContext instance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] A new Spark SQL command to merge small files [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#discussion_r1759699585
##########
extensions/spark/kyuubi-extension-spark-3-5/pom.xml:
##########
@@ -99,6 +99,14 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-avro_${scala.binary.version}</artifactId>
+ <version>${spark.version}</version>
+ <!-- Change this to match your Spark version -->
Review Comment:
nit remove comment
##########
extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/KyuubiSparkSQLExtension.scala:
##########
@@ -19,13 +19,15 @@ package org.apache.kyuubi.sql
import org.apache.spark.sql.{FinalStageResourceManager, InjectCustomResourceProfile, SparkSessionExtensions}
+import org.apache.kyuubi.sql.compact.CompactTableSparkStrategy
import org.apache.kyuubi.sql.watchdog.{ForcedMaxOutputRowsRule, KyuubiUnsupportedOperationsCheck, MaxScanStrategy}
// scalastyle:off line.size.limit
+
/**
* Depend on Spark SQL Extension framework, we can use this extension follow steps
* 1. move this jar into $SPARK_HOME/jars
- * 2. add config into `spark-defaults.conf`: `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`
+ * 2. add config into `spark-defaults.conf`: `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`
Review Comment:
nit indent
##########
extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/compact/CachePerformanceViewCommand.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.sql.compact
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.spark.sql.{Row, SparkInternalExplorer, SparkSession}
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.command.{DropTableCommand, LeafRunnableCommand}
+
+case class CachePerformanceViewCommand(
+ tableIdentifier: Seq[String],
+ performancePlan: LogicalPlan,
+ originalFileLocations: Seq[String],
+ options: CompactTableOption) extends LeafRunnableCommand {
+
+ override def innerChildren: Seq[QueryPlan[_]] = Seq(performancePlan)
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val dropViewCommand = DropTableCommand(
+ CompactTableUtils.getTableIdentifier(tableIdentifier),
+ ifExists = true,
+ isView = true,
+ purge = true)
+ dropViewCommand.run(sparkSession)
+
+ val speculation =
+ sparkSession.sparkContext.getConf.getBoolean("spark.speculation", defaultValue = false)
+ if (speculation) {
+ sparkSession.sparkContext.getConf.set("spark.speculation", "false")
Review Comment:
Not sure if set can take effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2360029615
we'd better talk about the syntax and make a final decision in the dev emails [dev@kyuubi.apache.org](mailto:dev@kyuubi.apache.org)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2357345783
> thank you @gabrywu , about the syntax, is there any reason to introduce `compact table` ? Why not use procedure like
>
> ```
> Call kyuubi.compact(table => 'xx', targetSizeInBytes => 'xxx', mode => 'xxx')
> ```
The answer is same to the question, why use procedure?
I created this command based on spark2.x when the call procedure was not the mainstream proposal.
And by the way, a call procedure is becoming a mainstream proposal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2359850318
For the syntax part, given Delta and Iceberg's dominance in the lakehouse market, I suggest following either Delta's `VACCUM` or Iceberg's `CALL` syntax.
Additional information:
- Kyuubi Spark extension already adopted Delta `ZORDER` syntax
- Spark 4.0 adopting the Iceberg `CALL` syntax, see SPARK-48781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2357303629
thank you @gabrywu , about the syntax, is there any reason to introduce `compact table` ? Why not use procedure like
```
Call kyuubi.compact(table => 'xx', targetSizeInBytes => 'xxx', mode => 'xxx')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2358107993
@cxzl25 what do you think of the `call kyuubi.compact` procedure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2359737743
@AngersZhuuuu can you help to review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2351642952
@cxzl25 @ulysses-you all tests pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Compact table [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6695:
URL: https://github.com/apache/kyuubi/pull/6695#issuecomment-2348434022
cc @ulysses-you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6691] A new Spark SQL command to merge small files [kyuubi]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu closed pull request #6695: [KYUUBI #6691] A new Spark SQL command to merge small files
URL: https://github.com/apache/kyuubi/pull/6695
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org