You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/04 16:20:02 UTC
[GitHub] [spark] CHENXCHEN opened a new pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files with SymlinkTextInputFormat And Fix Analyze for SymlinkTextInputFormat
CHENXCHEN opened a new pull request #35734:
URL: https://github.com/apache/spark/pull/35734
### What changes were proposed in this pull request?
1. Add support for reading ORC/Parquet files with SymlinkTextInputFormat table
2. Fix analyze table size for SymlinkTextInputFormat
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
1. Both trino(prestosql) and prestodb support reading ORC/Parquet, but Spark does not, Spark support other formats except ORC/Parquet format, and this PR add support for ORC/Parquet
2. As follows example, analyze the table size will be 100 instead of 19999, which affects the join optimization, and it will attempt to deliver a large table broadcast if `manifest` file size is less than `spark.sql.autoBroadcastJoinThreshold`, and after this PR, analyze the table size will be 19999
we have files:
```
size filepath
100 hdfs:///path/to/table/manifest
9999 hdfs:////path/to/other/part-1.parquet.orc
10000 hdfs://path/to/other/part-2.parquet.orc
content of `hdfs:///path/to/table/manifest` :
hdfs:///path/to/other/part-1.parquet.orc
hdfs:///path/to/other/part-2.parquet.orc
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHENXCHEN commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
CHENXCHEN commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1060228367
cc @cloud-fan could you help take a look when you have time? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35734:
URL: https://github.com/apache/spark/pull/35734#discussion_r820743467
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
##########
@@ -74,21 +74,44 @@ object CommandUtils extends Logging {
def calculateTotalSize(spark: SparkSession, catalogTable: CatalogTable): BigInt = {
val sessionState = spark.sessionState
val startTime = System.nanoTime()
+
+ val isSymlinkTable = SymlinkTextInputFormatUtil.isSymlinkTextFormat(catalogTable)
+ lazy val fs = new Path(catalogTable.location)
+ .getFileSystem(spark.sessionState.newHadoopConf())
val totalSize = if (catalogTable.partitionColumnNames.isEmpty) {
- calculateSingleLocationSize(sessionState, catalogTable.identifier,
- catalogTable.storage.locationUri)
+ if (isSymlinkTable) {
+ calculateMultipleLocationSizes(spark, catalogTable.identifier
+ , SymlinkTextInputFormatUtil.getSymlinkTableLocationPaths(fs, catalogTable.location)
+ ).sum
+ } else {
+ calculateSingleLocationSize(sessionState, catalogTable.identifier,
+ catalogTable.storage.locationUri)
+ }
} else {
// Calculate table size as a sum of the visible partitions. See SPARK-21079
val partitions = sessionState.catalog.listPartitions(catalogTable.identifier)
logInfo(s"Starting to calculate sizes for ${partitions.length} partitions.")
- val paths = partitions.map(_.storage.locationUri)
- calculateMultipleLocationSizes(spark, catalogTable.identifier, paths).sum
+ calculateMultipleLocationSizes(spark, catalogTable.identifier,
+ getPartitionPaths(partitions, isSymlinkTable, fs)).sum
}
logInfo(s"It took ${(System.nanoTime() - startTime) / (1000 * 1000)} ms to calculate" +
s" the total size for table ${catalogTable.identifier}.")
totalSize
}
+ def getPartitionPaths(partitions: Seq[CatalogTablePartition]
+ , isSymlinkTable: Boolean, fs: => FileSystem): Seq[Option[URI]] = {
Review comment:
```suggestion
def getPartitionPaths(
partitions: Seq[CatalogTablePartition],
isSymlinkTable: Boolean,
fs: => FileSystem): Seq[Option[URI]] = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #35734:
URL: https://github.com/apache/spark/pull/35734#discussion_r821279787
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##########
@@ -122,7 +122,16 @@ abstract class PartitioningAwareFileIndex(
// Directory does not exist, or has no children files
Nil
}
- PartitionDirectory(values, files)
+ // Check leaf files since they might be symlink targets
+ if (files.isEmpty) {
+ val status: Seq[FileStatus] = leafFiles.get(path) match {
Review comment:
> the real data file is not in the table root
Yea, but `leafFiles` are listed from table root, how symlink targets might be in leaf files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1059996101
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #35734:
URL: https://github.com/apache/spark/pull/35734#discussion_r821013673
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##########
@@ -122,7 +122,16 @@ abstract class PartitioningAwareFileIndex(
// Directory does not exist, or has no children files
Nil
}
- PartitionDirectory(values, files)
+ // Check leaf files since they might be symlink targets
+ if (files.isEmpty) {
+ val status: Seq[FileStatus] = leafFiles.get(path) match {
Review comment:
Are Symlink targets in leaf files? I think leaf files are listed from table root?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHENXCHEN commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files with SymlinkTextInputFormat And Fix Analyze for SymlinkTextInputFormat
Posted by GitBox <gi...@apache.org>.
CHENXCHEN commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1059324923
Could you please take a look and review this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1059722344
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35734:
URL: https://github.com/apache/spark/pull/35734#discussion_r820743765
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
##########
@@ -74,21 +74,44 @@ object CommandUtils extends Logging {
def calculateTotalSize(spark: SparkSession, catalogTable: CatalogTable): BigInt = {
val sessionState = spark.sessionState
val startTime = System.nanoTime()
+
+ val isSymlinkTable = SymlinkTextInputFormatUtil.isSymlinkTextFormat(catalogTable)
+ lazy val fs = new Path(catalogTable.location)
+ .getFileSystem(spark.sessionState.newHadoopConf())
val totalSize = if (catalogTable.partitionColumnNames.isEmpty) {
- calculateSingleLocationSize(sessionState, catalogTable.identifier,
- catalogTable.storage.locationUri)
+ if (isSymlinkTable) {
+ calculateMultipleLocationSizes(spark, catalogTable.identifier
+ , SymlinkTextInputFormatUtil.getSymlinkTableLocationPaths(fs, catalogTable.location)
+ ).sum
+ } else {
+ calculateSingleLocationSize(sessionState, catalogTable.identifier,
+ catalogTable.storage.locationUri)
+ }
} else {
// Calculate table size as a sum of the visible partitions. See SPARK-21079
val partitions = sessionState.catalog.listPartitions(catalogTable.identifier)
logInfo(s"Starting to calculate sizes for ${partitions.length} partitions.")
- val paths = partitions.map(_.storage.locationUri)
- calculateMultipleLocationSizes(spark, catalogTable.identifier, paths).sum
+ calculateMultipleLocationSizes(spark, catalogTable.identifier,
+ getPartitionPaths(partitions, isSymlinkTable, fs)).sum
}
logInfo(s"It took ${(System.nanoTime() - startTime) / (1000 * 1000)} ms to calculate" +
s" the total size for table ${catalogTable.identifier}.")
totalSize
}
+ def getPartitionPaths(partitions: Seq[CatalogTablePartition]
+ , isSymlinkTable: Boolean, fs: => FileSystem): Seq[Option[URI]] = {
+ partitions.flatMap { catalogTablePartition =>
+ if (isSymlinkTable) {
Review comment:
nit: 2 spaces indentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHENXCHEN commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
CHENXCHEN commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1059334617
cc @cloud-fan @gengliangwang @viirya @sadikovi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1061032987
Same feeling here. Not sure if `SymlinkTextInputFormat` is the only case with custom file listing behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHENXCHEN commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
CHENXCHEN commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1059906695
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1062492803
Agree with @cloud-fan that we don't have a good abstraction for this feature.
IMO, we can add a new method under `CatalogTable` and `CatalogTablePartition`, such as
```
getFileList(fileFormat: Stringļ¼ fs: FileSystem)
```
Then we can directly list files according to file format, when in all other place, we can just call this API. Then we don't need to judge if it's `SymlinkTextFormat` every where.
WDYT @cloud-fan @viirya @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHENXCHEN commented on a change in pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
CHENXCHEN commented on a change in pull request #35734:
URL: https://github.com/apache/spark/pull/35734#discussion_r821276608
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##########
@@ -122,7 +122,16 @@ abstract class PartitioningAwareFileIndex(
// Directory does not exist, or has no children files
Nil
}
- PartitionDirectory(values, files)
+ // Check leaf files since they might be symlink targets
+ if (files.isEmpty) {
+ val status: Seq[FileStatus] = leafFiles.get(path) match {
Review comment:
`SymlinkTextInputFormat` table specify the list of files for a table/partition based on the content of a text file.
As the example in the PR description shows, the real data file is not in the table root
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHENXCHEN commented on a change in pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
CHENXCHEN commented on a change in pull request #35734:
URL: https://github.com/apache/spark/pull/35734#discussion_r821288770
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##########
@@ -122,7 +122,16 @@ abstract class PartitioningAwareFileIndex(
// Directory does not exist, or has no children files
Nil
}
- PartitionDirectory(values, files)
+ // Check leaf files since they might be symlink targets
+ if (files.isEmpty) {
+ val status: Seq[FileStatus] = leafFiles.get(path) match {
Review comment:
At the partition table, we have the same behavior, there is also symbolic in the root directory of the partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1060758497
I think the problem today is we don't have a good abstraction for this feature at the framework level. This is a special hive file format that changes the behavior of file listing, while in Spark the `FileIndex` API assumes the file listing behavior is unrelated to the file format.
This PR simply adds special handling of `SymlinkTextInputFormat` in several places, and I'm OK with it if it's super hard to come up with a good abstraction for it, but we should give it a try first.
cc @viirya @dongjoon-hyun @AngersZhuuuu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1059722344
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHENXCHEN commented on pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
CHENXCHEN commented on pull request #35734:
URL: https://github.com/apache/spark/pull/35734#issuecomment-1059332808
This PR is a bit like [this](https://github.com/apache/spark/pull/29330) but is too old to support Spark 3.2.
In addition, this PR add a fix to the analysis table size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CHENXCHEN commented on a change in pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
CHENXCHEN commented on a change in pull request #35734:
URL: https://github.com/apache/spark/pull/35734#discussion_r824335808
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
##########
@@ -220,6 +233,8 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
if (paths.isEmpty) {
Seq(tablePath)
+ } else if (isSymlinkTextFormat) {
Review comment:
@AngersZhuuuu also used in here when `lazyPruningEnabled` = false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #35734: [SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #35734:
URL: https://github.com/apache/spark/pull/35734#discussion_r822228457
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
##########
@@ -204,10 +204,23 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
(options, None)
}
+ lazy val fs = tablePath.getFileSystem(sparkSession.sparkContext.hadoopConfiguration)
+ val isSymlinkTextFormat = SymlinkTextInputFormatUtil.isSymlinkTextFormat(relation.tableMeta)
+
+ val symlinkTargets = if (isSymlinkTextFormat) {
+ SymlinkTextInputFormatUtil.getTargetPathsFromSymlink(fs, tablePath)
Review comment:
When table is partitioned and `lazyPruningEnabled ` = false, this value is not used, I think we don't need to get this value earliear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org