You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2025/01/07 19:55:40 UTC
[PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
ArnavBalyan opened a new pull request, #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456
- Currently Scan is offloaded to Velox, however Velox does not support encrypted parquet files.
- This leads to a Velox side error and runtime failure. Added support to fallback in such cases.
- We attempt to read parquet footer for the root paths in scan operator, if decryption is detected, the scan fallsback
- This is behind a config and will not be enabled by default, users can enable scan fallback by passing `spark.gluten.sql.fallbackEncryptedParquet` true.
- Currently parquet 1.12 does not offer direct signal for checking encryption, it's available in 1.13 which can be used in the future if spark upgrades.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579421619
> Hi @jackylee-ch, spark 3.5 uses parquet 1.13. After parquet 1.13, there is a new field added to check for encryption, which can provide if the file is encrypted. However if we try to read the encrypted file footer, it throws ParquetCryptoRuntimeException. Could you please elaborate on why it may not work on 3.5? Thanks
You mean that no matter which Spark version we use, we can get `ParquetCryptoRuntimeException` if I try to read the encrypted footer?
BTW what would happen if the footer is not encrypted but the column is encrypted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2576148180
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579403712
> @Yohahaha The way to check encrypted file haven't been confirmed, I'm afraid this is not work on Spark 3.5
@jackylee-ch sorry, I missed your comments.
the check is disabled by default, it's safe to merge to main branch. if we verified it's not work in Spark 3.5, we can fix with a followup PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#discussion_r1907080739
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala:
##########
@@ -191,10 +191,35 @@ object VeloxBackendSettings extends BackendSettingsApi {
}
}
- validateScheme().orElse(validateFormat()) match {
- case Some(reason) => ValidationResult.failed(reason)
- case _ => ValidationResult.succeeded
+ def validateEncryption(): Option[String] = {
+
+ val encryptionValidationEnabled = GlutenConfig.getConf.enableEncryptedParquetFallback
Review Comment:
The config name is a little wird to me. maybe `parquetEncryptionValidationEnabled`?
##########
backends-velox/src/main/scala/org/apache/gluten/utils/ParquetMetadataUtils.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.utils
+
+import org.apache.gluten.extension.ValidationResult
+import org.apache.gluten.substrait.rel.LocalFilesNode.ReadFileFormat
+import org.apache.gluten.substrait.rel.LocalFilesNode.ReadFileFormat.ParquetReadFormat
+
+import org.apache.spark.util.SerializableConfiguration
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, LocatedFileStatus, Path, RemoteIterator}
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException
+import org.apache.parquet.hadoop.ParquetFileReader
+
+object ParquetMetadataUtils {
+
+ /**
+ * Validates whether Parquet encryption is enabled for the given paths.
+ *
+ * - If the file format is not Parquet, skip this check and return success.
+ * - If there is at least one Parquet file with encryption enabled, fail the validation.
+ *
+ * @param format
+ * File format, e.g., `ParquetReadFormat`
+ * @param rootPaths
+ * List of file paths to scan
+ * @param serializableHadoopConf
+ * Optional Hadoop configuration
+ * @return
+ * [[ValidationResult]] validation success or failure
+ */
+ def validateEncryption(
+ format: ReadFileFormat,
+ rootPaths: Seq[String],
+ serializableHadoopConf: Option[SerializableConfiguration]
+ ): ValidationResult = {
+ if (format != ParquetReadFormat || rootPaths.isEmpty) {
+ return ValidationResult.succeeded
+ }
+
+ val conf = serializableHadoopConf.map(_.value).getOrElse(new Configuration())
+
+ rootPaths.foreach {
+ rootPath =>
+ val fs = new Path(rootPath).getFileSystem(conf)
+ try {
+ val encryptionDetected =
+ checkForEncryptionWithLimit(fs, new Path(rootPath), conf, fileLimit = 10)
+ if (encryptionDetected) {
+ return ValidationResult.failed("Encrypted Parquet file detected.")
+ }
+ } catch {
+ case e: Exception =>
+ }
+ }
+ ValidationResult.succeeded
+ }
+
+ /**
+ * Check any Parquet file under the given path is encrypted using a recursive
+ * iterator. Only the first `fileLimit` files are processed for efficiency.
+ *
+ * @param fs
+ * FileSystem to use
+ * @param path
+ * Root path to check
+ * @param conf
+ * Hadoop configuration
+ * @param fileLimit
+ * Maximum number of files to inspect
+ * @return
+ * True if an encrypted file is detected, false otherwise
+ */
+ private def checkForEncryptionWithLimit(
+ fs: FileSystem,
+ path: Path,
+ conf: Configuration,
+ fileLimit: Int
+ ): Boolean = {
+
+ val filesIterator: RemoteIterator[LocatedFileStatus] = fs.listFiles(path, true)
+ var checkedFileCount = 0
+ while (filesIterator.hasNext && checkedFileCount < fileLimit) {
+ val fileStatus = filesIterator.next()
+ checkedFileCount += 1
+ try {
+ ParquetFileReader.readFooter(conf, fileStatus.getPath).toString
Review Comment:
Is it a better way to check encrypted metadata than use the Exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2576114406
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2578422050
Thanks for the review @Yohahaha addressed comments, could you please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579015279
```
[ERROR] /__w/incubator-gluten/incubator-gluten/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala:196: error: value getConf is not a member of object org.apache.gluten.config.GlutenConfig
[ERROR] val encryptionValidationEnabled = GlutenConfig.getConf.parquetEncryptionValidationEnabled
[ERROR] ^
[ERROR] one error found
[ERROR] exception compilation error occurred!!!
```
please rebase and fix conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#discussion_r1908201626
##########
backends-velox/src/main/scala/org/apache/gluten/utils/ParquetMetadataUtils.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.utils
+
+import org.apache.gluten.extension.ValidationResult
+import org.apache.gluten.substrait.rel.LocalFilesNode.ReadFileFormat
+import org.apache.gluten.substrait.rel.LocalFilesNode.ReadFileFormat.ParquetReadFormat
+
+import org.apache.spark.util.SerializableConfiguration
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, LocatedFileStatus, Path, RemoteIterator}
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException
+import org.apache.parquet.hadoop.ParquetFileReader
+
+object ParquetMetadataUtils {
+
+ /**
+ * Validates whether Parquet encryption is enabled for the given paths.
+ *
+ * - If the file format is not Parquet, skip this check and return success.
+ * - If there is at least one Parquet file with encryption enabled, fail the validation.
+ *
+ * @param format
+ * File format, e.g., `ParquetReadFormat`
+ * @param rootPaths
+ * List of file paths to scan
+ * @param serializableHadoopConf
+ * Optional Hadoop configuration
+ * @return
+ * [[ValidationResult]] validation success or failure
+ */
+ def validateEncryption(
+ format: ReadFileFormat,
+ rootPaths: Seq[String],
+ serializableHadoopConf: Option[SerializableConfiguration]
+ ): ValidationResult = {
+ if (format != ParquetReadFormat || rootPaths.isEmpty) {
+ return ValidationResult.succeeded
+ }
+
+ val conf = serializableHadoopConf.map(_.value).getOrElse(new Configuration())
+
+ rootPaths.foreach {
+ rootPath =>
+ val fs = new Path(rootPath).getFileSystem(conf)
+ try {
+ val encryptionDetected =
+ checkForEncryptionWithLimit(fs, new Path(rootPath), conf, fileLimit = 10)
+ if (encryptionDetected) {
+ return ValidationResult.failed("Encrypted Parquet file detected.")
+ }
+ } catch {
+ case e: Exception =>
+ }
+ }
+ ValidationResult.succeeded
+ }
+
+ /**
+ * Check any Parquet file under the given path is encrypted using a recursive
+ * iterator. Only the first `fileLimit` files are processed for efficiency.
+ *
+ * @param fs
+ * FileSystem to use
+ * @param path
+ * Root path to check
+ * @param conf
+ * Hadoop configuration
+ * @param fileLimit
+ * Maximum number of files to inspect
+ * @return
+ * True if an encrypted file is detected, false otherwise
+ */
+ private def checkForEncryptionWithLimit(
+ fs: FileSystem,
+ path: Path,
+ conf: Configuration,
+ fileLimit: Int
+ ): Boolean = {
+
+ val filesIterator: RemoteIterator[LocatedFileStatus] = fs.listFiles(path, true)
+ var checkedFileCount = 0
+ while (filesIterator.hasNext && checkedFileCount < fileLimit) {
+ val fileStatus = filesIterator.next()
+ checkedFileCount += 1
+ try {
+ ParquetFileReader.readFooter(conf, fileStatus.getPath).toString
Review Comment:
Spark 3.5 uses parquet 1.13, and I found it defined in thrift. This could work for Spark 3.5?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2578717307
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579448873
> > Hi @jackylee-ch, spark 3.5 uses parquet 1.13. After parquet 1.13, there is a new field added to check for encryption, which can provide if the file is encrypted. However if we try to read the encrypted file footer, it throws ParquetCryptoRuntimeException. Could you please elaborate on why it may not work on 3.5? Thanks
>
> @ArnavBalyan You mean that no matter which Spark version we use, we can get `ParquetCryptoRuntimeException` if I try to read the encrypted footer?
>
> BTW what would happen if the footer is not encrypted but the column is encrypted?
Sure let me add a follow up UT for 3.5, this feature is behind feature flag and verified for parquet 1.13. @jackylee-ch, it would depend how you are doing encryption in your setup. Typically the footer metadata will indicate encryption for newer versions of parquet.
> we do not need package parquet deps in Gluten I think, just do verification in shim module.
@Yohahaha, if we want to keep it backward compatible for parquet 1.12, and not use exception, then we will need a newer parquet inside Gluten regardless of the spark version, in that case, the check can hold true if there are future parquet upgrades inside spark. Using the shim layer we can probably do validation but still will use exception checking for 1.12.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2576179078
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2576114029
https://github.com/apache/incubator-gluten/issues/8455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2578697041
CI fails. can you fix format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579409308
@ArnavBalyan thanks for the update!
would you add UT in different spark shim module with a minimal encrypted parquet file?
we do not need package parquet deps in Gluten I think, just do verification in shim module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha merged PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579396531
> @Yohahaha The way to check encrypted file haven't been confirmed, I'm afraid this is not work on Spark 3.5
Hi @jackylee-ch, spark 3.5 uses parquet 1.15. After parquet 1.13, there is a new field added to check for encryption, which can provide if the file is encrypted. However if we try to read the encrypted file footer, it throws ParquetCryptoRuntimeException. Could you please elaborate on why it may not work on 3.5? Thanks
Alternatively, I was planning to shade parquet 1.14 and bring the shaded parquet as a packaged dependency in Gluten (to check the encryption), which will be a more elegant solution and work with multiple spark version. If there are no concerns with bringing in shaded parquet inside gluten, I'm happy to work on that implementation as well. I could not see such implementations inside Gluten so was a bit hesitant, althought parquet encryption seems to be a special case and could benefit from such a solution. Let me know what you think thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579515861
We can use different checker for different Spark version in shims, and for parquet 1.12, the exception is good to me if there is no other better idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#discussion_r1907670983
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala:
##########
@@ -191,10 +191,35 @@ object VeloxBackendSettings extends BackendSettingsApi {
}
}
- validateScheme().orElse(validateFormat()) match {
- case Some(reason) => ValidationResult.failed(reason)
- case _ => ValidationResult.succeeded
+ def validateEncryption(): Option[String] = {
+
+ val encryptionValidationEnabled = GlutenConfig.getConf.enableEncryptedParquetFallback
Review Comment:
Updated it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2576147256
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579435289
> @jackylee-ch I'm ok if you want to revert this PR with Spark 3.5 compatibility concern.
Current PR should work fine for Spark 3.2 and 3.3 for encrypted footer, not sure for other cases. If it doesn't support those cases, we need move `ParquetMetadataUtils` to shim layer and support other cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on code in PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#discussion_r1907018267
##########
shims/common/src/main/scala/org/apache/gluten/config/GlutenConfig.scala:
##########
@@ -2278,4 +2281,11 @@ object GlutenConfig {
"Otherwise, broadcast build relation will use onheap memory.")
.booleanConf
.createWithDefault(false)
+
+ val ENCRYPTED_PARQUET_FALLBACK_ENABLED =
+ buildStaticConf("spark.gluten.sql.fallbackEncryptedParquet")
Review Comment:
why use static conf?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#discussion_r1907672174
##########
backends-velox/src/main/scala/org/apache/gluten/utils/ParquetMetadataUtils.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.utils
+
+import org.apache.gluten.extension.ValidationResult
+import org.apache.gluten.substrait.rel.LocalFilesNode.ReadFileFormat
+import org.apache.gluten.substrait.rel.LocalFilesNode.ReadFileFormat.ParquetReadFormat
+
+import org.apache.spark.util.SerializableConfiguration
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, LocatedFileStatus, Path, RemoteIterator}
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException
+import org.apache.parquet.hadoop.ParquetFileReader
+
+object ParquetMetadataUtils {
+
+ /**
+ * Validates whether Parquet encryption is enabled for the given paths.
+ *
+ * - If the file format is not Parquet, skip this check and return success.
+ * - If there is at least one Parquet file with encryption enabled, fail the validation.
+ *
+ * @param format
+ * File format, e.g., `ParquetReadFormat`
+ * @param rootPaths
+ * List of file paths to scan
+ * @param serializableHadoopConf
+ * Optional Hadoop configuration
+ * @return
+ * [[ValidationResult]] validation success or failure
+ */
+ def validateEncryption(
+ format: ReadFileFormat,
+ rootPaths: Seq[String],
+ serializableHadoopConf: Option[SerializableConfiguration]
+ ): ValidationResult = {
+ if (format != ParquetReadFormat || rootPaths.isEmpty) {
+ return ValidationResult.succeeded
+ }
+
+ val conf = serializableHadoopConf.map(_.value).getOrElse(new Configuration())
+
+ rootPaths.foreach {
+ rootPath =>
+ val fs = new Path(rootPath).getFileSystem(conf)
+ try {
+ val encryptionDetected =
+ checkForEncryptionWithLimit(fs, new Path(rootPath), conf, fileLimit = 10)
+ if (encryptionDetected) {
+ return ValidationResult.failed("Encrypted Parquet file detected.")
+ }
+ } catch {
+ case e: Exception =>
+ }
+ }
+ ValidationResult.succeeded
+ }
+
+ /**
+ * Check any Parquet file under the given path is encrypted using a recursive
+ * iterator. Only the first `fileLimit` files are processed for efficiency.
+ *
+ * @param fs
+ * FileSystem to use
+ * @param path
+ * Root path to check
+ * @param conf
+ * Hadoop configuration
+ * @param fileLimit
+ * Maximum number of files to inspect
+ * @return
+ * True if an encrypted file is detected, false otherwise
+ */
+ private def checkForEncryptionWithLimit(
+ fs: FileSystem,
+ path: Path,
+ conf: Configuration,
+ fileLimit: Int
+ ): Boolean = {
+
+ val filesIterator: RemoteIterator[LocatedFileStatus] = fs.listFiles(path, true)
+ var checkedFileCount = 0
+ while (filesIterator.hasNext && checkedFileCount < fileLimit) {
+ val fileStatus = filesIterator.next()
+ checkedFileCount += 1
+ try {
+ ParquetFileReader.readFooter(conf, fileStatus.getPath).toString
Review Comment:
This is done to keep it backward compatible, spark 33 uses parquet 1.12, support is not there yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579367583
@Yohahaha The way to check encrypted file haven't been confirmed, I'm afraid this is not work on Spark 3.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579412856
@jackylee-ch I'm ok if you want to revert this PR with Spark 3.5 compatibility concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2578418652
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#discussion_r1907672960
##########
shims/common/src/main/scala/org/apache/gluten/config/GlutenConfig.scala:
##########
@@ -2278,4 +2281,11 @@ object GlutenConfig {
"Otherwise, broadcast build relation will use onheap memory.")
.booleanConf
.createWithDefault(false)
+
+ val ENCRYPTED_PARQUET_FALLBACK_ENABLED =
+ buildStaticConf("spark.gluten.sql.fallbackEncryptedParquet")
Review Comment:
updated, this config changing at runtime is an unlikely event, kept removed static to allow any such usecases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456#issuecomment-2579172889
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org