You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/23 22:57:21 UTC
[GitHub] [spark] Yaohua628 opened a new pull request, #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Yaohua628 opened a new pull request, #38777:
URL: https://github.com/apache/spark/pull/38777
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
A follow-up PR of https://github.com/apache/spark/pull/38683.
Apart from making `_metadata` struct not nullable, we should also make all fields inside of `_metadata` not nullable (`file_path`, `file_name`, `file_modification_time`, `file_size`, `row_index`).
### Why are the changes needed?
Consistent nullability behavior for everything
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
New UTs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1336748392
oh it conflicts with 3.3, @Yaohua628 can you open a backport PR? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yaohua628 commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
Yaohua628 commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1333389196
Addressed comments, thanks for taking a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yaohua628 commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
Yaohua628 commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1336286184
Thanks for the explanation, @ala! I am OK either way cc @cloud-fan @HeartSaVioR feel free to merge it if you think it is OK ^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38777:
URL: https://github.com/apache/spark/pull/38777#discussion_r1036655896
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -272,7 +273,7 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
case FileFormat.ROW_INDEX =>
fileFormatReaderGeneratedMetadataColumns
.find(_.name == FileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
- .get.withName(FileFormat.ROW_INDEX)
+ .get.withName(FileFormat.ROW_INDEX).withNullability(false)
Review Comment:
can we add a code comment to explain this nullability change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1332915287
I don't have context for that, sorry. I'm OK either way if the nullability of column is guaranteed to not fluctuate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1325830554
Let me see if there are further review comments today. I will merge this tomorrow if there is no outstanding comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1325844004
Ah OK, let's wait for feedback from @ala and ensure we make clear before merging it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1325769084
state store schema checker handles the compatibility for nullability. It does not only allow equality, but also allow the case when column for existing schema is nullable whereas column for new schema is non-nullable. So ensuring columns to be non-nullable would be OK for compatibility point of view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yaohua628 commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
Yaohua628 commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1332079823
@HeartSaVioR @ala some tests in `FileMetadataStructRowIndexSuite` are failed complaining:
```
java.io.IOException: Required column is missing in data file. Col: [_tmp_metadata_row_index]
[info] at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.checkColumn(VectorizedParquetRecordReader.java:375)
[info] at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.initializeInternal(VectorizedParquetRecordReader.java:349)
[info] at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:181)
```
I had a fix [here](https://github.com/apache/spark/pull/38777/commits/0f98dd97a3c84a10be6fd7f75468e5d62f17fe82) to resolve failures (basically: keep the internal `_tmp_metadata_row_index` nullable, but `_metadata.row_index` is still not null). I don't fully understand what happened internally, could you take a look? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
URL: https://github.com/apache/spark/pull/38777
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ala commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
ala commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1330730907
Sorry, I was on PTO/sick for a couple of days.
My idea was not to include the row index in `_metadata` for formats that cannot generate it. While we have _many_ Parquet readers, I believe all of them support row index generation right now, so that's not a problem.
For me the main concern might be that we might want to keep growing the `_metadata` struct (most recently: row id, cc @tomvanbussel, @juliuszsompolski). These new fields might not be immediately supported in all the readers (or if they are Databricks-internal, we might not want to support them in OSS readers at all). So while I think it's OK to make row index non-nullable, it would be an issue if we wanted to require that all `_metadata` fields are non-nullable in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1331589572
@Yaohua628 Could you please push a new empty commit or rebase to master branch to retrigger build? Let's make sure build is green before merging this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yaohua628 commented on a diff in pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
Yaohua628 commented on code in PR #38777:
URL: https://github.com/apache/spark/pull/38777#discussion_r1036931467
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -272,7 +272,10 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
case FileFormat.ROW_INDEX =>
fileFormatReaderGeneratedMetadataColumns
.find(_.name == FileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
- .get.withName(FileFormat.ROW_INDEX)
+ // Change the `_tmp_metadata_row_index` to `row_index`,
+ // and also change the nullability to not nullable,
+ // which is consistent with the nullability of `row_index` field
+ .get.withName(FileFormat.ROW_INDEX).withNullability(false)
Review Comment:
Thanks, Wenchen, I tried that before, but it failed many test cases in `FileMetadataStructRowIndexSuite`. See [this fix commit](https://github.com/apache/spark/pull/38777/commits/0f98dd97a3c84a10be6fd7f75468e5d62f17fe82) and [this comment](https://github.com/apache/spark/pull/38777#issuecomment-1332079823).
I don't have much context on the `row_index`, not sure what caused the issue, any idea? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1327384414
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1336549324
It seems OK to me as well but I'll lean on @cloud-fan on the decision as I'm not an expert on this subject.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1336747654
thanks, merging to master/3.3!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38777:
URL: https://github.com/apache/spark/pull/38777#discussion_r1036655709
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -234,7 +234,8 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
" is a reserved column name that cannot be read in combination with " +
s"${FileFormat.METADATA_NAME}.${FileFormat.ROW_INDEX} column.")
}
- Some(AttributeReference(FileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME, LongType)())
+ Some(AttributeReference(
+ FileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME, LongType)())
Review Comment:
unnecessary change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38777:
URL: https://github.com/apache/spark/pull/38777#discussion_r1036882653
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -272,7 +272,10 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
case FileFormat.ROW_INDEX =>
fileFormatReaderGeneratedMetadataColumns
.find(_.name == FileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
- .get.withName(FileFormat.ROW_INDEX)
+ // Change the `_tmp_metadata_row_index` to `row_index`,
+ // and also change the nullability to not nullable,
+ // which is consistent with the nullability of `row_index` field
+ .get.withName(FileFormat.ROW_INDEX).withNullability(false)
Review Comment:
shall we update `fileFormatReaderGeneratedMetadataColumns` to set nullablity as false?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ala commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
ala commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1333637318
Well, the issue seems to be that the vectorized reader recognizes the row index column as a "missing column" (aka. columns that are not read from the file, but instead populated by a higher layer in the reader). Since these are normally populated with nulls, it's a problem if the data type is non-nullable. https://github.com/apache/spark/blob/0f1c515179e5ed34aca27c51f500c26ca19cc748/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java#L372-L376
We could tweak this `if` condition to not throw on generate column/row index, or use the workaround you put in place already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yaohua628 commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
Yaohua628 commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1325752236
@cloud-fan @dongjoon-hyun @HeartSaVioR Sorry for the back and forth.
[The previous PR](https://github.com/apache/spark/pull/38683), we changed the `_metadata` to not null. And I just realized we probably should make all fields inside of the `_metadata` (`file_path`, `file_name`, `file_modification_time`, `file_size`, `row_index`) not null as well for consistency.
Please let me know WDYT. As @cloud-fan mentioned, it should be fine to write not-null data into a nullable column. But my only concern is this change might break the existing stateful streaming schema compatibility check?
Also, cc @ala to confirm `row_index` will always be not null for supported file formats (e.g. Parquet)
Thanks for all your help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yaohua628 commented on pull request #38777: [SPARK-41151][FOLLOW-UP][SQL] Keep built-in file _metadata fields nullable value consistent
Posted by GitBox <gi...@apache.org>.
Yaohua628 commented on PR #38777:
URL: https://github.com/apache/spark/pull/38777#issuecomment-1325843243
Thank you, Jungtaek! Also wanna confirm with @ala on nullability of `row_index`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org