You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/25 06:48:45 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method ` to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
LuciferYang opened a new pull request, #37654:
URL: https://github.com/apache/spark/pull/37654
### What changes were proposed in this pull request?
This pr is a refactor work, the main change is extract `ParquetUtils.prepareWrite` method to eliminate duplicate code in `ParquetFileFormat` and `ParquetWrite`.
### Why are the changes needed?
Eliminate duplicate code
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GitHub Actions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37654: [WIP][SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method ` to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37654:
URL: https://github.com/apache/spark/pull/37654#issuecomment-1226853913
cc @gengliangwang @cloud-fan Do you have time to help review? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
URL: https://github.com/apache/spark/pull/37654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37654: [WIP][SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method ` to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37654:
URL: https://github.com/apache/spark/pull/37654#discussion_r954569730
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -72,87 +69,9 @@ class ParquetFileFormat
job: Job,
options: Map[String, String],
dataSchema: StructType): OutputWriterFactory = {
- val parquetOptions = new ParquetOptions(options, sparkSession.sessionState.conf)
-
- val conf = ContextUtil.getConfiguration(job)
-
- val committerClass =
- conf.getClass(
- SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key,
- classOf[ParquetOutputCommitter],
- classOf[OutputCommitter])
-
- if (conf.get(SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key) == null) {
- logInfo("Using default output committer for Parquet: " +
- classOf[ParquetOutputCommitter].getCanonicalName)
- } else {
- logInfo("Using user defined output committer for Parquet: " + committerClass.getCanonicalName)
- }
-
- conf.setClass(
- SQLConf.OUTPUT_COMMITTER_CLASS.key,
- committerClass,
- classOf[OutputCommitter])
-
- // We're not really using `ParquetOutputFormat[Row]` for writing data here, because we override
- // it in `ParquetOutputWriter` to support appending and dynamic partitioning. The reason why
- // we set it here is to setup the output committer class to `ParquetOutputCommitter`, which is
- // bundled with `ParquetOutputFormat[Row]`.
- job.setOutputFormatClass(classOf[ParquetOutputFormat[Row]])
-
- ParquetOutputFormat.setWriteSupportClass(job, classOf[ParquetWriteSupport])
-
- // This metadata is useful for keeping UDTs like Vector/Matrix.
- ParquetWriteSupport.setSchema(dataSchema, conf)
-
- // Sets flags for `ParquetWriteSupport`, which converts Catalyst schema to Parquet
- // schema and writes actual rows to Parquet files.
- conf.set(
- SQLConf.PARQUET_WRITE_LEGACY_FORMAT.key,
- sparkSession.sessionState.conf.writeLegacyParquetFormat.toString)
-
- conf.set(
- SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key,
- sparkSession.sessionState.conf.parquetOutputTimestampType.toString)
-
- conf.set(
- SQLConf.PARQUET_FIELD_ID_WRITE_ENABLED.key,
- sparkSession.sessionState.conf.parquetFieldIdWriteEnabled.toString)
-
- conf.set(
- SQLConf.PARQUET_TIMESTAMP_NTZ_ENABLED.key,
- sparkSession.sessionState.conf.parquetTimestampNTZEnabled.toString)
-
- // Sets compression scheme
- conf.set(ParquetOutputFormat.COMPRESSION, parquetOptions.compressionCodecClassName)
-
- // SPARK-15719: Disables writing Parquet summary files by default.
- if (conf.get(ParquetOutputFormat.JOB_SUMMARY_LEVEL) == null
- && conf.get(ParquetOutputFormat.ENABLE_JOB_SUMMARY) == null) {
- conf.setEnum(ParquetOutputFormat.JOB_SUMMARY_LEVEL, JobSummaryLevel.NONE)
- }
-
- if (ParquetOutputFormat.getJobSummaryLevel(conf) != JobSummaryLevel.NONE
Review Comment:
This is the second difference:
in ParquetFileFormat:
```
if (ParquetOutputFormat.getJobSummaryLevel(conf) != JobSummaryLevel.NONE....
```
in ParquetWrite:
```
if (ParquetOutputFormat.getJobSummaryLevel(conf) == JobSummaryLevel.NONE
```
`!=` is used in `ParquetUtils.prepareWrite` because it exists earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37654:
URL: https://github.com/apache/spark/pull/37654#issuecomment-1260459690
Thanks @cloud-fan @sadikovi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on a diff in pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
sadikovi commented on code in PR #37654:
URL: https://github.com/apache/spark/pull/37654#discussion_r982881638
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -72,87 +69,9 @@ class ParquetFileFormat
job: Job,
options: Map[String, String],
dataSchema: StructType): OutputWriterFactory = {
- val parquetOptions = new ParquetOptions(options, sparkSession.sessionState.conf)
-
- val conf = ContextUtil.getConfiguration(job)
-
- val committerClass =
- conf.getClass(
- SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key,
- classOf[ParquetOutputCommitter],
- classOf[OutputCommitter])
-
- if (conf.get(SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key) == null) {
- logInfo("Using default output committer for Parquet: " +
- classOf[ParquetOutputCommitter].getCanonicalName)
- } else {
- logInfo("Using user defined output committer for Parquet: " + committerClass.getCanonicalName)
- }
-
- conf.setClass(
- SQLConf.OUTPUT_COMMITTER_CLASS.key,
- committerClass,
- classOf[OutputCommitter])
-
- // We're not really using `ParquetOutputFormat[Row]` for writing data here, because we override
- // it in `ParquetOutputWriter` to support appending and dynamic partitioning. The reason why
- // we set it here is to setup the output committer class to `ParquetOutputCommitter`, which is
- // bundled with `ParquetOutputFormat[Row]`.
- job.setOutputFormatClass(classOf[ParquetOutputFormat[Row]])
-
- ParquetOutputFormat.setWriteSupportClass(job, classOf[ParquetWriteSupport])
-
- // This metadata is useful for keeping UDTs like Vector/Matrix.
- ParquetWriteSupport.setSchema(dataSchema, conf)
-
- // Sets flags for `ParquetWriteSupport`, which converts Catalyst schema to Parquet
- // schema and writes actual rows to Parquet files.
- conf.set(
- SQLConf.PARQUET_WRITE_LEGACY_FORMAT.key,
- sparkSession.sessionState.conf.writeLegacyParquetFormat.toString)
-
- conf.set(
- SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key,
- sparkSession.sessionState.conf.parquetOutputTimestampType.toString)
-
- conf.set(
- SQLConf.PARQUET_FIELD_ID_WRITE_ENABLED.key,
- sparkSession.sessionState.conf.parquetFieldIdWriteEnabled.toString)
-
- conf.set(
- SQLConf.PARQUET_TIMESTAMP_NTZ_ENABLED.key,
- sparkSession.sessionState.conf.parquetTimestampNTZEnabled.toString)
-
- // Sets compression scheme
- conf.set(ParquetOutputFormat.COMPRESSION, parquetOptions.compressionCodecClassName)
-
- // SPARK-15719: Disables writing Parquet summary files by default.
- if (conf.get(ParquetOutputFormat.JOB_SUMMARY_LEVEL) == null
- && conf.get(ParquetOutputFormat.ENABLE_JOB_SUMMARY) == null) {
- conf.setEnum(ParquetOutputFormat.JOB_SUMMARY_LEVEL, JobSummaryLevel.NONE)
- }
-
- if (ParquetOutputFormat.getJobSummaryLevel(conf) != JobSummaryLevel.NONE
Review Comment:
Yes, DSv1 version is the correct one. We essentially disallow creating job summaries with an unknown output committer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37654:
URL: https://github.com/apache/spark/pull/37654#issuecomment-1263288488
friendly ping @cloud-fan , should we merge this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #37654:
URL: https://github.com/apache/spark/pull/37654#issuecomment-1260343604
@sadikovi can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37654: [WIP][SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method ` to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37654:
URL: https://github.com/apache/spark/pull/37654#discussion_r954569730
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -72,87 +69,9 @@ class ParquetFileFormat
job: Job,
options: Map[String, String],
dataSchema: StructType): OutputWriterFactory = {
- val parquetOptions = new ParquetOptions(options, sparkSession.sessionState.conf)
-
- val conf = ContextUtil.getConfiguration(job)
-
- val committerClass =
- conf.getClass(
- SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key,
- classOf[ParquetOutputCommitter],
- classOf[OutputCommitter])
-
- if (conf.get(SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key) == null) {
- logInfo("Using default output committer for Parquet: " +
- classOf[ParquetOutputCommitter].getCanonicalName)
- } else {
- logInfo("Using user defined output committer for Parquet: " + committerClass.getCanonicalName)
- }
-
- conf.setClass(
- SQLConf.OUTPUT_COMMITTER_CLASS.key,
- committerClass,
- classOf[OutputCommitter])
-
- // We're not really using `ParquetOutputFormat[Row]` for writing data here, because we override
- // it in `ParquetOutputWriter` to support appending and dynamic partitioning. The reason why
- // we set it here is to setup the output committer class to `ParquetOutputCommitter`, which is
- // bundled with `ParquetOutputFormat[Row]`.
- job.setOutputFormatClass(classOf[ParquetOutputFormat[Row]])
-
- ParquetOutputFormat.setWriteSupportClass(job, classOf[ParquetWriteSupport])
-
- // This metadata is useful for keeping UDTs like Vector/Matrix.
- ParquetWriteSupport.setSchema(dataSchema, conf)
-
- // Sets flags for `ParquetWriteSupport`, which converts Catalyst schema to Parquet
- // schema and writes actual rows to Parquet files.
- conf.set(
- SQLConf.PARQUET_WRITE_LEGACY_FORMAT.key,
- sparkSession.sessionState.conf.writeLegacyParquetFormat.toString)
-
- conf.set(
- SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key,
- sparkSession.sessionState.conf.parquetOutputTimestampType.toString)
-
- conf.set(
- SQLConf.PARQUET_FIELD_ID_WRITE_ENABLED.key,
- sparkSession.sessionState.conf.parquetFieldIdWriteEnabled.toString)
-
- conf.set(
- SQLConf.PARQUET_TIMESTAMP_NTZ_ENABLED.key,
- sparkSession.sessionState.conf.parquetTimestampNTZEnabled.toString)
-
- // Sets compression scheme
- conf.set(ParquetOutputFormat.COMPRESSION, parquetOptions.compressionCodecClassName)
-
- // SPARK-15719: Disables writing Parquet summary files by default.
- if (conf.get(ParquetOutputFormat.JOB_SUMMARY_LEVEL) == null
- && conf.get(ParquetOutputFormat.ENABLE_JOB_SUMMARY) == null) {
- conf.setEnum(ParquetOutputFormat.JOB_SUMMARY_LEVEL, JobSummaryLevel.NONE)
- }
-
- if (ParquetOutputFormat.getJobSummaryLevel(conf) != JobSummaryLevel.NONE
Review Comment:
This is the second difference:
in ParquetFileFormat:
```
if (ParquetOutputFormat.getJobSummaryLevel(conf) != JobSummaryLevel.NONE....
```
in ParquetWrite:
```
if (ParquetOutputFormat.getJobSummaryLevel(conf) == JobSummaryLevel.NONE...
```
`!=` is used in `ParquetUtils.prepareWrite` because it exists earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37654: [WIP][SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method ` to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37654:
URL: https://github.com/apache/spark/pull/37654#discussion_r954566232
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -72,87 +69,9 @@ class ParquetFileFormat
job: Job,
options: Map[String, String],
dataSchema: StructType): OutputWriterFactory = {
- val parquetOptions = new ParquetOptions(options, sparkSession.sessionState.conf)
-
- val conf = ContextUtil.getConfiguration(job)
-
- val committerClass =
- conf.getClass(
- SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key,
- classOf[ParquetOutputCommitter],
- classOf[OutputCommitter])
-
- if (conf.get(SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key) == null) {
- logInfo("Using default output committer for Parquet: " +
- classOf[ParquetOutputCommitter].getCanonicalName)
- } else {
- logInfo("Using user defined output committer for Parquet: " + committerClass.getCanonicalName)
- }
-
- conf.setClass(
- SQLConf.OUTPUT_COMMITTER_CLASS.key,
- committerClass,
- classOf[OutputCommitter])
-
- // We're not really using `ParquetOutputFormat[Row]` for writing data here, because we override
- // it in `ParquetOutputWriter` to support appending and dynamic partitioning. The reason why
- // we set it here is to setup the output committer class to `ParquetOutputCommitter`, which is
- // bundled with `ParquetOutputFormat[Row]`.
- job.setOutputFormatClass(classOf[ParquetOutputFormat[Row]])
-
- ParquetOutputFormat.setWriteSupportClass(job, classOf[ParquetWriteSupport])
-
- // This metadata is useful for keeping UDTs like Vector/Matrix.
- ParquetWriteSupport.setSchema(dataSchema, conf)
-
- // Sets flags for `ParquetWriteSupport`, which converts Catalyst schema to Parquet
- // schema and writes actual rows to Parquet files.
- conf.set(
- SQLConf.PARQUET_WRITE_LEGACY_FORMAT.key,
- sparkSession.sessionState.conf.writeLegacyParquetFormat.toString)
-
- conf.set(
- SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key,
- sparkSession.sessionState.conf.parquetOutputTimestampType.toString)
-
- conf.set(
- SQLConf.PARQUET_FIELD_ID_WRITE_ENABLED.key,
- sparkSession.sessionState.conf.parquetFieldIdWriteEnabled.toString)
-
- conf.set(
- SQLConf.PARQUET_TIMESTAMP_NTZ_ENABLED.key,
Review Comment:
This is the first difference:
`SQLConf.PARQUET_TIMESTAMP_NTZ_ENABLED.key` exists only in `ParquetFileFormat.prepareWrite`but not in `ParquetWrite.prepareWrite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang closed pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
LuciferYang closed pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
URL: https://github.com/apache/spark/pull/37654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37654:
URL: https://github.com/apache/spark/pull/37654#issuecomment-1269472892
Thanks @sadikovi @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on a diff in pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
sadikovi commented on code in PR #37654:
URL: https://github.com/apache/spark/pull/37654#discussion_r982881822
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -72,87 +69,9 @@ class ParquetFileFormat
job: Job,
options: Map[String, String],
dataSchema: StructType): OutputWriterFactory = {
- val parquetOptions = new ParquetOptions(options, sparkSession.sessionState.conf)
-
- val conf = ContextUtil.getConfiguration(job)
-
- val committerClass =
- conf.getClass(
- SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key,
- classOf[ParquetOutputCommitter],
- classOf[OutputCommitter])
-
- if (conf.get(SQLConf.PARQUET_OUTPUT_COMMITTER_CLASS.key) == null) {
- logInfo("Using default output committer for Parquet: " +
- classOf[ParquetOutputCommitter].getCanonicalName)
- } else {
- logInfo("Using user defined output committer for Parquet: " + committerClass.getCanonicalName)
- }
-
- conf.setClass(
- SQLConf.OUTPUT_COMMITTER_CLASS.key,
- committerClass,
- classOf[OutputCommitter])
-
- // We're not really using `ParquetOutputFormat[Row]` for writing data here, because we override
- // it in `ParquetOutputWriter` to support appending and dynamic partitioning. The reason why
- // we set it here is to setup the output committer class to `ParquetOutputCommitter`, which is
- // bundled with `ParquetOutputFormat[Row]`.
- job.setOutputFormatClass(classOf[ParquetOutputFormat[Row]])
-
- ParquetOutputFormat.setWriteSupportClass(job, classOf[ParquetWriteSupport])
-
- // This metadata is useful for keeping UDTs like Vector/Matrix.
- ParquetWriteSupport.setSchema(dataSchema, conf)
-
- // Sets flags for `ParquetWriteSupport`, which converts Catalyst schema to Parquet
- // schema and writes actual rows to Parquet files.
- conf.set(
- SQLConf.PARQUET_WRITE_LEGACY_FORMAT.key,
- sparkSession.sessionState.conf.writeLegacyParquetFormat.toString)
-
- conf.set(
- SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key,
- sparkSession.sessionState.conf.parquetOutputTimestampType.toString)
-
- conf.set(
- SQLConf.PARQUET_FIELD_ID_WRITE_ENABLED.key,
- sparkSession.sessionState.conf.parquetFieldIdWriteEnabled.toString)
-
- conf.set(
- SQLConf.PARQUET_TIMESTAMP_NTZ_ENABLED.key,
Review Comment:
I think we just forgot to add it to DSv2. Thanks for fixing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #37654:
URL: https://github.com/apache/spark/pull/37654#issuecomment-1269468573
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sadikovi commented on pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
sadikovi commented on PR #37654:
URL: https://github.com/apache/spark/pull/37654#issuecomment-1260445669
Yes, I will take a look, thanks for letting me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37654: [SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37654:
URL: https://github.com/apache/spark/pull/37654#issuecomment-1261594105
Thanks @sadikovi ~ rebase to keep the code up to date, let's wait for GA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org