You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/12/14 16:41:16 UTC
[GitHub] [hudi] pengzhiwei2018 opened a new pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
pengzhiwei2018 opened a new pull request #2334:
URL: https://github.com/apache/hudi/pull/2334
…e hoodie table schema
add test case
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
Fix the bug when write a int type to the double type field in table, an exception throw out.
## Brief change log
Add inputSchema in HoodieWriteHandle for parse data in input generic record.
Add tableSchema in HoodieWriteHandle for read and write data in the table.
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- Added TestCOWDataSource#testWriteWithCompatibleType to verify the change
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 closed pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 closed pull request #2334:
URL: https://github.com/apache/hudi/pull/2334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (2e68c09) into [master](https://codecov.io/gh/apache/hudi/commit/38b9264dd041db955ffa4c0f10273e1b5fa21cfb?el=desc) (38b9264) will **decrease** coverage by `42.16%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
=============================================
- Coverage 52.20% 10.04% -42.17%
+ Complexity 2659 48 -2611
=============================================
Files 335 52 -283
Lines 14981 1852 -13129
Branches 1505 223 -1282
=============================================
- Hits 7821 186 -7635
+ Misses 6535 1653 -4882
+ Partials 625 13 -612
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `10.04% <ø> (-59.62%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [312 more](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#discussion_r552740514
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
##########
@@ -53,47 +52,70 @@
private static final Logger LOG = LogManager.getLogger(HoodieWriteHandle.class);
- protected final Schema writerSchema;
- protected final Schema writerSchemaWithMetafields;
+ /**
+ * The table schema is the schema of the table which used to read/write record from table.
+ */
+ protected final Schema tableSchema;
+ /**
+ * The table schema with meta fields.
+ */
+ protected final Schema tableSchemaWithMetaFields;
+ /**
+ * The input schema is the input data schema which used to parse data from incoming record.
+ */
+ protected final Schema inputSchema;
+ /**
+ * The input shema with meta fields.
+ */
+ protected final Schema inputSchemaWithMetaFields;
+
protected HoodieTimer timer;
protected final WriteStatus writeStatus;
protected final String partitionPath;
protected final String fileId;
protected final String writeToken;
protected final TaskContextSupplier taskContextSupplier;
- public HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath,
- String fileId, HoodieTable<T, I, K, O> hoodieTable, TaskContextSupplier taskContextSupplier) {
- this(config, instantTime, partitionPath, fileId, hoodieTable,
- getWriterSchemaIncludingAndExcludingMetadataPair(config), taskContextSupplier);
- }
+ /**
+ *
+ * @param config the write config
+ * @param instantTime the instance time
+ * @param partitionPath the partition path
+ * @param fileId the file id
+ * @param hoodieTable the hoodie table
+ * @param schemaOption the option schema specified for HoodieBootstrapHandle
+ * @param taskContextSupplier
+ */
+ protected HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath,
+ String fileId, HoodieTable<T, I, K, O> hoodieTable,
+ Option<Schema> schemaOption,
+ TaskContextSupplier taskContextSupplier) {
- protected HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath, String fileId,
- HoodieTable<T, I, K, O> hoodieTable, Pair<Schema, Schema> writerSchemaIncludingAndExcludingMetadataPair,
- TaskContextSupplier taskContextSupplier) {
super(config, instantTime, hoodieTable);
this.partitionPath = partitionPath;
this.fileId = fileId;
- this.writerSchema = writerSchemaIncludingAndExcludingMetadataPair.getKey();
- this.writerSchemaWithMetafields = writerSchemaIncludingAndExcludingMetadataPair.getValue();
+ // Here the hoodieTable#config may not equal to the config field, we use the 'config' from the
+ // construct method
+ this.tableSchema = schemaOption.orElseGet(() -> hoodieTable.getTableSchema(config, false));
Review comment:
Yes @nsivabalan ,the `schemaOption` is only used for `HoodieBootstrapHandle` to pass it's schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-799071055
> @pengzhiwei2018 first of all, thanks for these great contributions.
>
> Wrt inputSchema vs writeSchema, I actually feel writeSchema already stands for inputSchema, input is what is being written, right? We can probably just leave it as is. and introduce new `tableSchema` variables as you have in the `HoodieWriteHandle` class.?
>
> Like someone else pointed out as well, so far, we are using read and write schemas consistently. Love to not introduce a new input schema, unless its absolutely necessary .
Hi @vinothchandar ,thanks for your reply on this issue.
Yes, in most case ,the `writeSchema` is the same with the `inputSchema` which can stands for the `inputSchema` . But in the case in this PR (test case in TestCOWDataSource) we write the table twice:
First, we write a "id: long" to the table. The input schema is "a:long", the table schema is "a:long".
Second, we write a "id:int" to the table. The input schema is "a:int", but the table schema is "a:long" as the previous write. The write schema should be the same with the table schema, or else an Exception would throw out which is the problem we want to solve in this PR.
So in this case, we need to distinguish the difference between the `inputSchema` and `writeSchema`. The `inputSchema` is the incoming records's schema, but the `writeSchema` is always the `tableSchema`.
- The `inputSchema` is used to parser the record from the incoming data.
- The `tableSchema` is used to write and read record from the table. When we want to write or read record to the table, we use the `tableSchema`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-751355407
> @pengzhiwei2018 : sorry I am not following the problem here. Can you throw some light.
> Schema evolution is throwing an exception w/ Hudi, i.e a field's data type evolved to a compatible data type and hudi fails to accommodate it? is that the problem.
> If yes, can you check #2350. there is a bug in schema compatibility check and the patch should fix it. If not, can you elaborate what exactly is the problem.
> For eg: use-case given at #2063 succeeds with #2350
Hi @nsivabalan, When I write a `long` value to the `double` type field in the table, It can pass the `TableSchemaResolver.isSchemaCompatible`,However it failed in the write stage.
I found the problem is that the schema (e.g `double`) used in write & read table are the same with the input schema(e.g. `long`). I fixed this problem by distinguishing the two schemas.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (b58bf4d) into [master](https://codecov.io/gh/apache/hudi/commit/38b9264dd041db955ffa4c0f10273e1b5fa21cfb?el=desc) (38b9264) will **decrease** coverage by `42.16%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
=============================================
- Coverage 52.20% 10.04% -42.17%
+ Complexity 2659 48 -2611
=============================================
Files 335 52 -283
Lines 14981 1852 -13129
Branches 1505 223 -1282
=============================================
- Hits 7821 186 -7635
+ Misses 6535 1653 -4882
+ Partials 625 13 -612
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `10.04% <ø> (-59.62%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [312 more](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-812021796
oh, didn't realize default schema compatibility check is false. I assume if I enable schema compatibility check, double to int evolution is likely to fail. So, not sure what this PR tries to achieve. At least from my local test run, integer to double worked for me.
https://gist.github.com/nsivabalan/91f12109e0fe1ca9749ff5290c946778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
prashantwason commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-806413911
So to rephrase the description this solves the case where the input-data has a compatible field (int) to be written to the table (long field). Can this issue not be solved at the input record level by converting the "int" data into the "long" before writing into HUDI?
hoodieTable.getTableSchema() always returns the "latest" schema which is the schema used for the last HoodieWriteClient (saved into commit instants). So when the "int" based RDD is written the table schema will no long have a "long" field. When this table schema is used to read an older file in the table (merge during updatehandle) then the reading should fail as a long (from parquet) cannot be converted to an int (from schema). This is actually a backward incompatible schema change and hence is not allowed by HUDI.
@pengzhiwei2018 Can you add a test to verify my hypothesis? In your existing test in TestCOWDataSource, can you write a long to the table in the next write? Also, can you read all the data using the "int" schema, even the older records which contain a long?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-799071055
> @pengzhiwei2018 first of all, thanks for these great contributions.
>
> Wrt inputSchema vs writeSchema, I actually feel writeSchema already stands for inputSchema, input is what is being written, right? We can probably just leave it as is. and introduce new `tableSchema` variables as you have in the `HoodieWriteHandle` class.?
>
> Like someone else pointed out as well, so far, we are using read and write schemas consistently. Love to not introduce a new input schema, unless its absolutely necessary .
Hi @vinothchandar ,thanks for your reply on this issue.
Yes, in most case ,the `writeSchema` is the same with the `inputSchema` which can stands for the `inputSchema` . But in the case in this PR, we write the table twice:
First, we write a "id: long" to the table. The input schema is "a:long", the table schema is "a:long".
Second, we write a "id:int" to the table. The input schema is "a:int", but the table schema is "a:long" as the previous write. The write schema should be the same with the table schema, or else an Exception would throw out which is the problem we want to solve in this PR.
So in this case, we need to distinguish the difference between the `inputSchema` and `writeSchema`. The `inputSchema` is the incoming records's schema, but the `writeSchema` is always the `tableSchema`.
**Here's the summary**
- The `inputSchema` is used to parser the record from the incoming data.
- The `tableSchema` is used to write and read record from the table. When we want to write or read record to the table, we use the `tableSchema`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (79422fe) into [master](https://codecov.io/gh/apache/hudi/commit/b593f1062931a4d017ae8bd7dd42e47a8873a39f?el=desc) (b593f10) will **decrease** coverage by `40.22%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
=============================================
- Coverage 50.27% 10.04% -40.23%
+ Complexity 2990 48 -2942
=============================================
Files 410 52 -358
Lines 18406 1852 -16554
Branches 1885 223 -1662
=============================================
- Hits 9253 186 -9067
+ Misses 8395 1653 -6742
+ Partials 758 13 -745
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `10.04% <ø> (-59.62%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [386 more](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (2e086b9) into [master](https://codecov.io/gh/apache/hudi/commit/236d1b0dec409efb9c7a9febd260b1060da6a9c1?el=desc) (236d1b0) will **decrease** coverage by `41.93%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
=============================================
- Coverage 52.34% 10.40% -41.94%
+ Complexity 2631 48 -2583
=============================================
Files 329 51 -278
Lines 14808 1787 -13021
Branches 1484 213 -1271
=============================================
- Hits 7751 186 -7565
+ Misses 6442 1588 -4854
+ Partials 615 13 -602
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `10.40% <ø> (-59.71%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [306 more](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (2e68c09) into [master](https://codecov.io/gh/apache/hudi/commit/38b9264dd041db955ffa4c0f10273e1b5fa21cfb?el=desc) (38b9264) will **decrease** coverage by `42.16%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
=============================================
- Coverage 52.20% 10.04% -42.17%
+ Complexity 2659 48 -2611
=============================================
Files 335 52 -283
Lines 14981 1852 -13129
Branches 1505 223 -1282
=============================================
- Hits 7821 186 -7635
+ Misses 6535 1653 -4882
+ Partials 625 13 -612
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `10.04% <ø> (-59.62%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [312 more](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (908bb9e) into [master](https://codecov.io/gh/apache/hudi/commit/5ff8e88d58c2a6efa1046716e11dca0d61cdf872?el=desc) (5ff8e88) will **increase** coverage by `0.01%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
============================================
+ Coverage 50.28% 50.29% +0.01%
Complexity 2991 2991
============================================
Files 410 410
Lines 18406 18406
Branches 1885 1885
============================================
+ Hits 9256 9258 +2
+ Misses 8392 8391 -1
+ Partials 758 757 -1
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `37.28% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.94% <0.00%> (+0.02%)` | `0.00 <0.00> (ø)` | |
| hudiflink | `10.20% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.35% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `62.81% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `48.61% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `65.30% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.65% <ø> (ø)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/common/table/TableSchemaResolver.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL1RhYmxlU2NoZW1hUmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `79.68% <0.00%> (+1.56%)` | `26.00% <0.00%> (ø%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-754400800
> https://github.com/apache/hudi/blob/master/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java#L70
Hi @n3nash ,Thanks for your suggestion.
This PR only covers the part where HUDi writes the data, not the part where hudi reads the data. The introduced `inputSchema/tableSchema` is only used for parser the incoming record and read&write data in table.which is much different from the `writer/reader schema` in RealtimeCompactedRecordReader.
And I will describe the changes in more detail in the **Brief change log** part of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (908bb9e) into [master](https://codecov.io/gh/apache/hudi/commit/5ff8e88d58c2a6efa1046716e11dca0d61cdf872?el=desc) (5ff8e88) will **decrease** coverage by `40.24%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
=============================================
- Coverage 50.28% 10.04% -40.25%
+ Complexity 2991 48 -2943
=============================================
Files 410 52 -358
Lines 18406 1852 -16554
Branches 1885 223 -1662
=============================================
- Hits 9256 186 -9070
+ Misses 8392 1653 -6739
+ Partials 758 13 -745
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `10.04% <ø> (-59.62%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [386 more](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#discussion_r551485287
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/AbstractMergeHelper.java
##########
@@ -103,7 +103,8 @@ protected GenericRecord transformRecordBasedOnNewSchema(GenericDatumReader<Gener
}
return new MergingIterator<>(reader.getRecordIterator(readSchema), bootstrapReader.getRecordIterator(bootstrapReadSchema),
- (inputRecordPair) -> HoodieAvroUtils.stitchRecords(inputRecordPair.getLeft(), inputRecordPair.getRight(), mergeHandle.getWriterSchemaWithMetafields()));
Review comment:
Can you revert this change ? Doesn't look like you are changing anything here ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-751308433
@pengzhiwei2018 : sorry I am not following the problem here. Can you throw some light.
Schema evolution is throwing an exception w/ Hudi, i.e a field's data type evolved to a compatible data type and hudi fails to accommodate it? is that the problem.
If yes, can you check https://github.com/apache/hudi/pull/2350. there is a bug in schema compatibility check and the patch should fix it. If not, can you elaborate what exactly is the problem.
For eg: use-case given at https://github.com/apache/hudi/issues/2063 succeeds with https://github.com/apache/hudi/pull/2350
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-812019464
just to clarify. some schema evolution just for types are working fine w/ hoodie. for eg: integer to double works fine. Problem is that, double to int is where the issue is. I am not sure why schema compatibility check does not fail this evolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-754400800
Hi @n3nash ,Thanks for your suggestion.
This PR only covers the part where HUDi writes the data, not the part where hudi reads the data. The introduced `inputSchema/tableSchema` is only used for parser the incoming record and read&write data in table.which is much different from the `writer/reader schema` in RealtimeCompactedRecordReader.
And I have update the describe the changes in more detail in the **Brief change log** part for better to understand. You take a look again when you have time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#discussion_r551698632
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
##########
@@ -53,47 +52,70 @@
private static final Logger LOG = LogManager.getLogger(HoodieWriteHandle.class);
- protected final Schema writerSchema;
- protected final Schema writerSchemaWithMetafields;
+ /**
+ * The table schema is the schema of the table which used to read/write record from table.
+ */
+ protected final Schema tableSchema;
+ /**
+ * The table schema with meta fields.
+ */
+ protected final Schema tableSchemaWithMetaFields;
+ /**
+ * The input schema is the input data schema which used to parse data from incoming record.
+ */
+ protected final Schema inputSchema;
+ /**
+ * The input shema with meta fields.
Review comment:
Thanks for your correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-812516130
Yeah, I did verify by enabling schema compatability check. it will fail if we try to evolve a field from double to int.
```
scala> dfFromData5.write.format("hudi").
| options(getQuickstartWriteConfigs).
| option(PRECOMBINE_FIELD_OPT_KEY, "preComb").
| option(RECORDKEY_FIELD_OPT_KEY, "rowId").
| option(PARTITIONPATH_FIELD_OPT_KEY, "partitionId").
| option("hoodie.index.type","SIMPLE").
| option(TABLE_NAME, tableName).
| option("hoodie.avro.schema.validate","true").
| mode(Append).
| save(basePath)
org.apache.hudi.exception.HoodieUpsertException: Failed upsert schema compatibility check.
at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:629)
at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:152)
at org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:214)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:186)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:677)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:677)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:677)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:286)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:272)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:230)
... 72 elided
Caused by: org.apache.hudi.exception.HoodieException: Failed schema compatibility check for writerSchema :{"type":"record","name":"hudi_trips_cow_record","namespace":"hoodie.hudi_trips_cow","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"rowId","type":["string","null"]},{"name":"partitionId","type":["string","null"]},{"name":"preComb","type":["long","null"]},{"name":"name","type":["string","null"]},{"name":"versionId","type":["string","null"]},{"name":"doubleToInt","type":["int","null"]}]}, table schema :{"type":"record","name":"hudi_trips_cow_record","namespace":"hoodie.hudi_trips_cow","fields":[{"name":"_hoodie_co
mmit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"rowId","type":["string","null"]},{"name":"partitionId","type":["string","null"]},{"name":"preComb","type":["long","null"]},{"name":"name","type":["string","null"]},{"name":"versionId","type":["string","null"]},{"name":"doubleToInt","type":["double","null"]}]}, base path :file:/tmp/hudi_trips_cow
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:621)
at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:627)
... 97 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#discussion_r593862114
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java
##########
@@ -152,9 +155,9 @@ public void write() {
final String key = keyIterator.next();
HoodieRecord<T> record = recordMap.get(key);
if (useWriterSchema) {
- write(record, record.getData().getInsertValue(writerSchemaWithMetafields));
+ write(record, record.getData().getInsertValue(inputSchemaWithMetaFields));
Review comment:
I actually disagree completely :) . Leaking such call hierarchy into a lower level class will lead to more confusions, if say one more code path uses this code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#discussion_r552744172
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java
##########
@@ -93,18 +93,14 @@ public HoodieMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTab
init(fileId, this.partitionPath, dataFileToBeMerged);
}
- @Override
- public Schema getWriterSchemaWithMetafields() {
- return writerSchemaWithMetafields;
- }
-
public Schema getWriterSchema() {
- return writerSchema;
+ return tableSchema;
Review comment:
It seems reasonable!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#discussion_r551485414
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapRecordConsumer.java
##########
@@ -36,10 +36,12 @@ public BootstrapRecordConsumer(HoodieBootstrapHandle bootstrapHandle) {
this.bootstrapHandle = bootstrapHandle;
}
+ @SuppressWarnings("unchecked")
@Override
protected void consumeOneRecord(HoodieRecord record) {
try {
- bootstrapHandle.write(record, record.getData().getInsertValue(bootstrapHandle.getWriterSchemaWithMetafields()));
+ bootstrapHandle.write(record, record.getData()
Review comment:
Can you revert this change ? Doesn't look like you're changing anything here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (a0dc8c7) into [master](https://codecov.io/gh/apache/hudi/commit/38b9264dd041db955ffa4c0f10273e1b5fa21cfb?el=desc) (38b9264) will **decrease** coverage by `42.16%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
=============================================
- Coverage 52.20% 10.04% -42.17%
+ Complexity 2659 48 -2611
=============================================
Files 335 52 -283
Lines 14981 1852 -13129
Branches 1505 223 -1282
=============================================
- Hits 7821 186 -7635
+ Misses 6535 1653 -4882
+ Partials 625 13 -612
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `10.04% <ø> (-59.62%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [312 more](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-754142222
@pengzhiwei2018 @nsivabalan Is there a different way to resolve this issue ? The writer/reader schema is baked into many parts of the code, even on the reader side, see example here -> https://github.com/apache/hudi/blob/master/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java#L70
To be able to remove the concept of writer into input requires more changes in the code and also requires us to understand what does the readerSchema mean in this scenario.
I'm OK if you want to introduce another transient field called `tableSchema` and compare the writer (in your case inputSchema) against the tableSchema to throw the exception, but keep the concepts of writer/reader intact. If you want to propose changing them, I think putting a 1 pager about how writer/reader schema maps to your new suggested way (input/table) will help to understand how to reason about schemas going forward to avoid any regressions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-799071055
> @pengzhiwei2018 first of all, thanks for these great contributions.
>
> Wrt inputSchema vs writeSchema, I actually feel writeSchema already stands for inputSchema, input is what is being written, right? We can probably just leave it as is. and introduce new `tableSchema` variables as you have in the `HoodieWriteHandle` class.?
>
> Like someone else pointed out as well, so far, we are using read and write schemas consistently. Love to not introduce a new input schema, unless its absolutely necessary .
Hi @vinothchandar ,thanks for your reply on this issue.
Yes, in most case ,the `writeSchema` is the same with the `inputSchema` which can stands for the `inputSchema` . But in the case in this PR, we write the table twice:
First, we write a "id: long" to the table. The input schema is "a:long", the table schema is "a:long".
Second, we write a "id:int" to the table. The input schema is "a:int", but the table schema is "a:long" as the previous write. The write schema should be the same with the table schema, or else an Exception would throw out which is the problem we want to solve in this PR.
So in this case, we need to distinguish the difference between the `inputSchema` and `writeSchema`. The `inputSchema` is the incoming records's schema, but the `writeSchema` is always the `tableSchema`.
- The `inputSchema` is used to parser the record from the incoming data.
- The `tableSchema` is used to write and read record from the table. When we want to write or read record to the table, we use the `tableSchema`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-812496550
> https://gist.github.com/nsivabalan/91f12109e0fe1ca9749ff5290c946778
Hi @nsivabalan , I have take a review for your test code. First you write a "int" to the table, so the table schema type is "int". Then you write a "double" to the table, so the table schema become to "double". The table schema changed from "int" to "double". I think this is more reasonable.
In my original idea, I think the first write schema(e.g. "int") is the table schema forever. The incoming records after that should be compatible with the origin table schema(e.g. "int"). This is this PR wants to solve. I can understand more clearly now. The table schema should be change to a more generic type (e.g. from "int" to "double"), but not always be the first write schema.
So I can close this PR now. Thanks @nsivabalan for correct me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#discussion_r550891438
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java
##########
@@ -152,9 +155,9 @@ public void write() {
final String key = keyIterator.next();
HoodieRecord<T> record = recordMap.get(key);
if (useWriterSchema) {
- write(record, record.getData().getInsertValue(writerSchemaWithMetafields));
+ write(record, record.getData().getInsertValue(inputSchemaWithMetaFields));
Review comment:
minor. not exactly related to this patch. But can we rename "useWriterSchema" to something like "isCompactor" so that its explicit and comprehensible.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
##########
@@ -53,47 +52,70 @@
private static final Logger LOG = LogManager.getLogger(HoodieWriteHandle.class);
- protected final Schema writerSchema;
- protected final Schema writerSchemaWithMetafields;
+ /**
+ * The table schema is the schema of the table which used to read/write record from table.
+ */
+ protected final Schema tableSchema;
+ /**
+ * The table schema with meta fields.
+ */
+ protected final Schema tableSchemaWithMetaFields;
+ /**
+ * The input schema is the input data schema which used to parse data from incoming record.
+ */
+ protected final Schema inputSchema;
+ /**
+ * The input shema with meta fields.
Review comment:
minor. typo. "schema"
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -574,9 +574,8 @@ private void validateSchema() throws HoodieUpsertException, HoodieInsertExceptio
Schema writerSchema;
boolean isValid;
try {
- TableSchemaResolver schemaUtil = new TableSchemaResolver(getMetaClient());
writerSchema = HoodieAvroUtils.createHoodieWriteSchema(config.getSchema());
- tableSchema = HoodieAvroUtils.createHoodieWriteSchema(schemaUtil.getTableAvroSchemaWithoutMetadataFields());
+ tableSchema = getTableSchema(config, true);
Review comment:
isn't the 2nd arg supposed to be false?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieBootstrapHandle.java
##########
@@ -38,8 +38,7 @@
public HoodieBootstrapHandle(HoodieWriteConfig config, String commitTime, HoodieTable<T, I, K, O> hoodieTable,
String partitionPath, String fileId, TaskContextSupplier taskContextSupplier) {
super(config, commitTime, hoodieTable, partitionPath, fileId,
- Pair.of(HoodieAvroUtils.RECORD_KEY_SCHEMA,
- HoodieAvroUtils.addMetadataFields(HoodieAvroUtils.RECORD_KEY_SCHEMA)), taskContextSupplier);
Review comment:
@bvaradar : can you confirm that this change looks good.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java
##########
@@ -93,18 +93,14 @@ public HoodieMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTab
init(fileId, this.partitionPath, dataFileToBeMerged);
}
- @Override
- public Schema getWriterSchemaWithMetafields() {
- return writerSchemaWithMetafields;
- }
-
public Schema getWriterSchema() {
- return writerSchema;
+ return tableSchema;
Review comment:
Feel we could rename this method to getReaderSchema() since this is used only at one place
```
bootstrapReadSchema = mergeHandle.getWriterSchema();
```
And we assign it to a var to store read schema anyways.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
##########
@@ -53,47 +52,70 @@
private static final Logger LOG = LogManager.getLogger(HoodieWriteHandle.class);
- protected final Schema writerSchema;
- protected final Schema writerSchemaWithMetafields;
+ /**
+ * The table schema is the schema of the table which used to read/write record from table.
+ */
+ protected final Schema tableSchema;
Review comment:
thanks for this change. Definitely looks more readable now and avoids any confusion
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
##########
@@ -53,47 +52,70 @@
private static final Logger LOG = LogManager.getLogger(HoodieWriteHandle.class);
- protected final Schema writerSchema;
- protected final Schema writerSchemaWithMetafields;
+ /**
+ * The table schema is the schema of the table which used to read/write record from table.
+ */
+ protected final Schema tableSchema;
+ /**
+ * The table schema with meta fields.
+ */
+ protected final Schema tableSchemaWithMetaFields;
+ /**
+ * The input schema is the input data schema which used to parse data from incoming record.
+ */
+ protected final Schema inputSchema;
+ /**
+ * The input shema with meta fields.
+ */
+ protected final Schema inputSchemaWithMetaFields;
+
protected HoodieTimer timer;
protected final WriteStatus writeStatus;
protected final String partitionPath;
protected final String fileId;
protected final String writeToken;
protected final TaskContextSupplier taskContextSupplier;
- public HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath,
- String fileId, HoodieTable<T, I, K, O> hoodieTable, TaskContextSupplier taskContextSupplier) {
- this(config, instantTime, partitionPath, fileId, hoodieTable,
- getWriterSchemaIncludingAndExcludingMetadataPair(config), taskContextSupplier);
- }
+ /**
+ *
+ * @param config the write config
+ * @param instantTime the instance time
+ * @param partitionPath the partition path
+ * @param fileId the file id
+ * @param hoodieTable the hoodie table
+ * @param schemaOption the option schema specified for HoodieBootstrapHandle
+ * @param taskContextSupplier
+ */
+ protected HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath,
+ String fileId, HoodieTable<T, I, K, O> hoodieTable,
+ Option<Schema> schemaOption,
+ TaskContextSupplier taskContextSupplier) {
- protected HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath, String fileId,
- HoodieTable<T, I, K, O> hoodieTable, Pair<Schema, Schema> writerSchemaIncludingAndExcludingMetadataPair,
- TaskContextSupplier taskContextSupplier) {
super(config, instantTime, hoodieTable);
this.partitionPath = partitionPath;
this.fileId = fileId;
- this.writerSchema = writerSchemaIncludingAndExcludingMetadataPair.getKey();
- this.writerSchemaWithMetafields = writerSchemaIncludingAndExcludingMetadataPair.getValue();
+ // Here the hoodieTable#config may not equal to the config field, we use the 'config' from the
+ // construct method
+ this.tableSchema = schemaOption.orElseGet(() -> hoodieTable.getTableSchema(config, false));
Review comment:
trying to confirm my understanding. Only for bootstrap code path, we need the table schema to be initialized w/ incoming Schema and can't rely on hoodieTable.getTableSchema(). If not, we should not be looking at schemaOption only to set tableSchema(in every other call paths).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#discussion_r551698427
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java
##########
@@ -152,9 +155,9 @@ public void write() {
final String key = keyIterator.next();
HoodieRecord<T> record = recordMap.get(key);
if (useWriterSchema) {
- write(record, record.getData().getInsertValue(writerSchemaWithMetafields));
+ write(record, record.getData().getInsertValue(inputSchemaWithMetaFields));
Review comment:
Nice suggestion！`isCompactor` well be better understanding.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (a0dc8c7) into [master](https://codecov.io/gh/apache/hudi/commit/38b9264dd041db955ffa4c0f10273e1b5fa21cfb?el=desc) (38b9264) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
============================================
+ Coverage 52.20% 52.22% +0.02%
- Complexity 2659 2660 +1
============================================
Files 335 335
Lines 14981 14981
Branches 1505 1505
============================================
+ Hits 7821 7824 +3
+ Misses 6535 6532 -3
Partials 625 625
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `38.83% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `54.80% <ø> (+0.03%)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.29% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `65.30% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.65% <ø> (ø)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ache/hudi/common/fs/inline/InMemoryFileSystem.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL2lubGluZS9Jbk1lbW9yeUZpbGVTeXN0ZW0uamF2YQ==) | `89.65% <0.00%> (+10.34%)` | `16.00% <0.00%> (+1.00%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-799071055
> @pengzhiwei2018 first of all, thanks for these great contributions.
>
> Wrt inputSchema vs writeSchema, I actually feel writeSchema already stands for inputSchema, input is what is being written, right? We can probably just leave it as is. and introduce new `tableSchema` variables as you have in the `HoodieWriteHandle` class.?
>
> Like someone else pointed out as well, so far, we are using read and write schemas consistently. Love to not introduce a new input schema, unless its absolutely necessary .
Hi @vinothchandar ,thanks for your reply on this issue.
Yes, in most case ,the `writeSchema` is the same with the `inputSchema` which can stands for the `inputSchema` . But in the case in this PR (test case in [TestCOWDataSource](https://github.com/apache/hudi/pull/2334/files#diff-9429f5bc432f70ea4801e306dd817416b76e6ab68d41a278e222c989ce5c9824)) we write the table twice:
First, we write a "id: long" to the table. The input schema is "a:long", the table schema is "a:long".
Second, we write a "id:int" to the table. The input schema is "a:int", but the table schema is "a:long" as the previous write. The write schema should be the same with the table schema, or else an Exception would throw out which is the problem we want to solve in this PR.
So in this case, we need to distinguish the difference between the `inputSchema` and `writeSchema`. The `inputSchema` is the incoming records's schema, but the `writeSchema` is always the `tableSchema`.
- The `inputSchema` is used to parser the record from the incoming data.
- The `tableSchema` is used to write and read record from the table. When we want to write or read record to the table, we use the `tableSchema`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-754400800
> https://github.com/apache/hudi/blob/master/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java#L70
Hi @n3nash ,Thanks for your suggestion.
This PR only covers the part where HUDi writes the data, not the part where hudi reads the data. The introduced `inputSchema/tableSchema` is only used for parser the incoming record and read&write data in table.which is much different from the `writer/reader schema` in RealtimeCompactedRecordReader.
And I will describe the changes in more detail in the **Brief change log** part for better to understand.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#discussion_r551699383
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -574,9 +574,8 @@ private void validateSchema() throws HoodieUpsertException, HoodieInsertExceptio
Schema writerSchema;
boolean isValid;
try {
- TableSchemaResolver schemaUtil = new TableSchemaResolver(getMetaClient());
writerSchema = HoodieAvroUtils.createHoodieWriteSchema(config.getSchema());
- tableSchema = HoodieAvroUtils.createHoodieWriteSchema(schemaUtil.getTableAvroSchemaWithoutMetadataFields());
+ tableSchema = getTableSchema(config, true);
Review comment:
Hi @nsivabalan ,here `createHoodieWriteSchema` will add the meta fields to the schema, so the `tableSchema` should also have the meta fields here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-812021796
oh, didn't realize default schema compatibility check is false. I assume if I enable schema compatibility check, double to int evolution is likely to fail. So, not sure what this PR tries to achieve. At least from my local test run, integer to double worked for me.
https://gist.github.com/nsivabalan/91f12109e0fe1ca9749ff5290c946778
this is COW btw. haven't tested w/ MOR.
@pengzhiwei2018 : can you please clarify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging [#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (bbb604a) into [master](https://codecov.io/gh/apache/hudi/commit/e926c1a45ca95fa1911f6f88a0577554f2797760?el=desc) (e926c1a) will **decrease** coverage by `41.04%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2334 +/- ##
============================================
- Coverage 50.73% 9.68% -41.05%
+ Complexity 3064 48 -3016
============================================
Files 419 53 -366
Lines 18797 1930 -16867
Branches 1922 230 -1692
============================================
- Hits 9536 187 -9349
+ Misses 8485 1730 -6755
+ Partials 776 13 -763
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.68% <ø> (-59.75%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [395 more](https://codecov.io/gh/apache/hudi/pull/2334/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org