You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/12/29 07:28:40 UTC
[GitHub] [flink] saLeox opened a new pull request, #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
saLeox opened a new pull request, #21149:
URL: https://github.com/apache/flink/pull/21149
## What is the purpose of the change
From the improvement [FLINK-23715](https://issues.apache.org/jira/browse/FLINK-23715), Flink use a collection named `unknownFieldsIndices` to track the nonexistent fields, and it is kept inside the `ParquetVectorizedInputFormat`, and applied to all parquet files under given path.
However, some fields may only be nonexistent in some of the historical parquet files, while exist in latest ones. And based on `unknownFieldsIndices`, flink will always skip these fields, even thought they are existing in the later parquets. As a result, the value of all fields will become empty when they are nonexistent in some historical parquet files.
This PR is going to make `unknownFieldsIndices` separate for each ParquetReader.
## Brief change log
- *Make `unknownFieldsIndices` as a member variable of ParquetReader, which is an inner class of ParquetVectorizedInputFormat*
## Verifying this change
This change added tests and can be verified as follows:
- *Added junit test that validates `unknownFieldsIndices` can differ from each parquets*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): ( no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1311252558
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1367123546
Hi @Tartarus0zm , thanks for your comments!
For the rule constraints, when reading each parquet split, there is an existing rule in Flink, to check whether a column from requested schema is missing from the file schema and if the field type is matched, for detail please refer to `ParquetReader::createReader` and `ParquetReader::checkColumn` in Flink. Our change will not break the rule.
In addition, it's a good point to keep the engine behavior consistent. I check the implementation in Spark. It's known that there is `mergeSchema` option when reading, and it will merge the schema from all specified parquet files, to achieve the schema evolution. After that, inside each parquet reader, they also have a collection called `missingColumns` to collect missing columns. For the missing column, the whole vector will be set as null, instead of throwing error and skipping it in schema. For your reference, Please check the implementation of the method `VectorizedParquetRecordReader::checkColumn` and `ParquetColumnVector::ParquetColumnVector` in the Spark. In that sense, this change will help user achieve schema evolution in Flink, as same as what they can do with Spark.
Please let me know if there is any concern or further clarification needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21149: [FLINK-29527][format] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1290315815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0342870aba81924a7a6b6c142c986ae6e79a7901",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0342870aba81924a7a6b6c142c986ae6e79a7901",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0342870aba81924a7a6b6c142c986ae6e79a7901 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Tartarus0zm commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
Tartarus0zm commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1363774890
@saLeox Thanks for contribution.
In our company, there are strict usage rules, so there is no such thing as multiple schema files in the same partition directory.
The changes is fine, but I think it's best to have rule constraints, I'm not sure what the behavior in hive/spark is like, I think it's best to keep the engine behavior consistent, otherwise it will bring additional interpretation costs.
What do you think? @saLeox @luoyuxia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] luoyuxia commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
luoyuxia commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1346338199
@saLeox Thanks for explanation. Let's wait the feedback from @Tartarus0zm .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] luoyuxia commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1420190347
@saLeox Thanks for your investigation.
Won't it means it'll be more reasonable that we also introuce a option for that?
@Tartarus0zm I'm a little of curious about why your company has strict usage rules. Is there any bad for multiple schema files in the same partition directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][format] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1293043228
@lirui-apache please help take a look, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox closed pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
saLeox closed pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
URL: https://github.com/apache/flink/pull/21149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tisonkun commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1665047919
Thanks for your contribution and all reviews!
Merging...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #21149:
URL: https://github.com/apache/flink/pull/21149#discussion_r1102396477
##########
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetVectorizedInputFormat.java:
##########
@@ -123,7 +122,8 @@ public ParquetReader createReader(final Configuration config, final SplitT split
FilterCompat.Filter filter = getFilter(hadoopConfig.conf());
List<BlockMetaData> blocks = filterRowGroups(filter, footer.getBlocks(), fileSchema);
- MessageType requestedSchema = clipParquetSchema(fileSchema);
+ HashSet<Integer> unknownFieldsIndices = new HashSet<>();
Review Comment:
```suggestion
Set<Integer> unknownFieldsIndices = new HashSet<>();
```
nit: Generally, it's good practice to use interfaces instead of classes in declarations.
##########
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetVectorizedInputFormat.java:
##########
@@ -309,6 +311,8 @@ private class ParquetReader implements BulkFormat.Reader<T> {
private final MessageType requestedSchema;
+ private final HashSet<Integer> unknownFieldsIndices;
Review Comment:
```suggestion
private final Set<Integer> unknownFieldsIndices;
```
nit: again, interfaces should be prefered.
##########
flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormatTest.java:
##########
@@ -294,6 +294,80 @@ void testProjectionReadUnknownField(int rowGroupSize) throws IOException {
});
}
+ @ParameterizedTest
+ @MethodSource("parameters")
+ void testProjectionReadUnknownFieldAcrossFiles(int rowGroupSize) throws IOException {
+ int number = 1000;
+ List<RowData> records = new ArrayList<>(number);
+ for (int i = 0; i < number; i++) {
+ Integer v = i;
+ records.add(newRow(v));
+ }
+ // create parquet files for both legacy and new
+ // f7 and f99 don’t exist in legacy parquet file
+ // f99 doesn't exist in new parquet file
+ // create legacy parquet file
+ RowType legacyParquetType =
+ new RowType(
+ IntStream.range(0, 6) // assume f7 not exist in legacy parquet file.
Review Comment:
```suggestion
IntStream.range(0, 6) // assume f7 and following columns do not exist in legacy parquet file.
```
Am I missing something here or should is the comment not reflecting the statement? :thinking: Don't we only create a `RowType` for the first 6 columns? :thinking:
##########
flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormatTest.java:
##########
@@ -294,6 +294,80 @@ void testProjectionReadUnknownField(int rowGroupSize) throws IOException {
});
}
+ @ParameterizedTest
+ @MethodSource("parameters")
+ void testProjectionReadUnknownFieldAcrossFiles(int rowGroupSize) throws IOException {
+ int number = 1000;
Review Comment:
What's the intention behind having such a large testdata set? Is it because we iterating over multiple group sizes that are provided through the `parameter()` test data method source? If the `1000` was selected because of that, it would be worth creating a static final field `MAX_ROW_GROUP_SIZE` and use this here as well. That would make it clearer why the value `1000` was selected.
##########
flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormatTest.java:
##########
@@ -294,6 +294,80 @@ void testProjectionReadUnknownField(int rowGroupSize) throws IOException {
});
}
+ @ParameterizedTest
+ @MethodSource("parameters")
Review Comment:
Out of curiosity: What do we gain by having so much test data in terms of `rowGroupSize`? Isn't the test here the same for a groupSize of 10 and 20 for instance which means we're not gaining anything from the test runs? Or am I missing something here?
##########
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetVectorizedInputFormat.java:
##########
@@ -170,7 +171,8 @@ public boolean isSplittable() {
}
/** Clips `parquetSchema` according to `fieldNames`. */
- private MessageType clipParquetSchema(GroupType parquetSchema) {
+ private MessageType clipParquetSchema(
+ GroupType parquetSchema, Set<Integer> unknownFieldsIndices) {
Review Comment:
```suggestion
GroupType parquetSchema, Collection<Integer> unknownFieldsIndices) {
```
nit: We're only using `add` in this method. Therefore, we could make the method signature even more relaxed.
##########
flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormatTest.java:
##########
@@ -294,6 +294,80 @@ void testProjectionReadUnknownField(int rowGroupSize) throws IOException {
});
}
+ @ParameterizedTest
+ @MethodSource("parameters")
+ void testProjectionReadUnknownFieldAcrossFiles(int rowGroupSize) throws IOException {
+ int number = 1000;
+ List<RowData> records = new ArrayList<>(number);
+ for (int i = 0; i < number; i++) {
+ Integer v = i;
+ records.add(newRow(v));
+ }
Review Comment:
```suggestion
final List<RowData> records =
IntStream.range(0, 1000).mapToObj(this::newRow).collect(Collectors.toList());
```
nit: your proposal works fine. I just got puzzled by the `Integer v = i` line which is obsolete. It made me think how to shorten the overall block. ...`newRow` could be even made `static`.
##########
flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormatTest.java:
##########
@@ -294,6 +294,80 @@ void testProjectionReadUnknownField(int rowGroupSize) throws IOException {
});
}
+ @ParameterizedTest
+ @MethodSource("parameters")
+ void testProjectionReadUnknownFieldAcrossFiles(int rowGroupSize) throws IOException {
+ int number = 1000;
+ List<RowData> records = new ArrayList<>(number);
+ for (int i = 0; i < number; i++) {
+ Integer v = i;
+ records.add(newRow(v));
+ }
+ // create parquet files for both legacy and new
+ // f7 and f99 don’t exist in legacy parquet file
+ // f99 doesn't exist in new parquet file
+ // create legacy parquet file
+ RowType legacyParquetType =
+ new RowType(
+ IntStream.range(0, 6) // assume f7 not exist in legacy parquet file.
+ .mapToObj(i -> ROW_TYPE.getFields().get(i))
+ .collect(Collectors.toList()));
+ Path legacyParquetPath =
+ createTempParquetFile(
+ new File(folder, "legacy"), legacyParquetType, records, rowGroupSize);
+ // create new parquet file
+ Path newParquetPath = createTempParquetFile(new File(folder, "new"), records, rowGroupSize);
+
+ // test reader
+ String[] readerColumnNames = new String[] {"f7", "f2", "f4", "f99"};
+ LogicalType[] fieldTypes =
+ new LogicalType[] {
+ new DoubleType(), new TinyIntType(), new IntType(), new VarCharType()
+ };
Review Comment:
```suggestion
final LogicalType[] fieldTypes = Arrays.stream(readerColumnNames)
.map(ROW_TYPE::getFieldIndex)
.map(ROW_TYPE::getTypeAt)
.toArray(LogicalType[]::new);
```
Could we use that instead? This would make the test more robust against test data changes (i.e. the data types are dynamically derived from the static `ROW_TYPE` instead of hard-coded). WDYT?
##########
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetVectorizedInputFormat.java:
##########
@@ -335,10 +339,12 @@ private class ParquetReader implements BulkFormat.Reader<T> {
private ParquetReader(
ParquetFileReader reader,
MessageType requestedSchema,
+ HashSet<Integer> unknownFieldsIndices,
Review Comment:
```suggestion
Set<Integer> unknownFieldsIndices,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tisonkun commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1425526377
> because I'm not that familiar with the code
Interesting. I found reviewers from GitHub suggestions. Then it's GitHub's fault, lol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] 1996fanrui commented on pull request #21149: [FLINK-29527][format] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
1996fanrui commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1291765606
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1343829748
Hi @luoyuxia, glad to hear the feedback and the idea behind design!
The case that mix different schema files in one single partition will happen when we use date as partition key in the ODS layer in the data warehouse. When the upstream products change their schema, we have to adapt to the schema change as well, and better not overwrite the schema of existing historical data.
Further more, in our Flink job, we consume the ODS data, and treat it as unbounded data in the streaming execution modes by using chain function `monitorContinuously`.
The `ParquetReader` is the underlying implementation that used to read data for the above mode, and it's expected to handle the schema evolution.
Hope it can make more sense, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] luoyuxia commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
luoyuxia commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1336860164
@saLeox Thanks for contribution.
I had a quick look about the changes. These changes looks reasonable to me.
@Tartarus0zm Could you please have a look again to check these changes since you are the author of [FLINK-23715]?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] luoyuxia commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
luoyuxia commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1341903988
@saLeox The code changes looks good to me. But after an offline discussion with @Tartarus0zm, I have some other conerns about this pr.
> However, some fields may only be nonexistent in some of the historical parquet files, while exist in latest ones.
In which case will it happen? In FLINK-23715, we make `unknownFieldsIndices` a memeber in `ParquetVectorizedInputFormat` instead of `ParquetReader ` by design. The basic background is that we usually add column in dataware house, so it may well happen that some historical partitions miss some columns, then we make it can be still read. And also, considering the normal case is that we first add column and write a **new** partition, so in here, we assume single one partition will have single one schema. There shouldn't mix some files with one shema and some files with another schema in one partition.
So, in here, we choose a more conservative strategy which will throw exception instead of reading miss columns as null value when reading one single partition.
Personally, I think the changes are fine, and the case you descibe will happen. But I'm wondering is it the normal case in your production env and is it the best practice that mix different schema files in one single partition.
If so, I aggree it should be merged to Flink's codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1367098878
> @saLeox Thanks for contribution. In our company, there are strict usage rules, so there is no such thing as multiple schema files in the same partition directory. The changes is fine, but I think it's best to have rule constraints, I'm not sure what the behavior in hive/spark is like, I think it's best to keep the engine behavior consistent, otherwise it will bring additional interpretation costs. What do you think? @saLeox @luoyuxia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by "saLeox (via GitHub)" <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1422086364
@luoyuxia Thanks for your comment, it sounds make sense to have similar option to control whether if merge schema, I will try to add them later, and notify here if make any progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by "saLeox (via GitHub)" <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1434133679
Hi @XComp, thanks for your comment! and it is quite useful and make the code more lean and robust.
I modify code accordingly and please help take a look again.
For the size of test dataset, I try to reduce them from 1000 to 100, and only change them in the functions that I am familiar with, to minimise the impact on other unit test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1311251003
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][format] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1292864583
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][format] Make unknownFieldsIndices work for single ParquetReader
Posted by GitBox <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1291388833
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tisonkun merged pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun merged PR #21149:
URL: https://github.com/apache/flink/pull/21149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by "saLeox (via GitHub)" <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1647629511
> @saLeox Thanks for your investigation. Won't it means it'll be more reasonable that we also introuce a option for that? @Tartarus0zm I'm a little of curious about why your company has strict usage rules. Is there any bad for multiple schema files in the same partition directory.
Could you help take a look at this PR again, as it's pending for a long while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] saLeox commented on pull request #21149: [FLINK-29527][formats/parquet] Make unknownFieldsIndices work for single ParquetReader
Posted by "saLeox (via GitHub)" <gi...@apache.org>.
saLeox commented on PR #21149:
URL: https://github.com/apache/flink/pull/21149#issuecomment-1647354136
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org