You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/28 02:46:21 UTC
[GitHub] [hudi] TengHuo opened a new pull request, #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
TengHuo opened a new pull request, #7307:
URL: https://github.com/apache/hudi/pull/7307
Trouble shooting detail in issue: https://github.com/apache/hudi/issues/7284
### Change Logs
* Add a new parameter in method `HoodieBaseRelation#convertToAvroSchema` for generating a aualified name according to the table name for Avro schema
* Use the new API `AvroConversionUtils#convertStructTypeToAvroSchema` in `HoodieBaseRelation#convertToAvroSchema` for struct schema to avro schema converting
* Add a new test case `TestMorTable` for verifying the issue is fixed
### Impact
No public API changed.
### Risk level (write none, low medium or high below)
low
### Documentation Update
No doc or configuration changed.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1403015540
@TengHuo can you please rebase to the latest master and verify whether fix is still relevant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1328462186
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "53575f2200794996c14749d78afce6535d5272ac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "53575f2200794996c14749d78afce6535d5272ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53575f2200794996c14749d78afce6535d5272ac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo closed pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo closed pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
URL: https://github.com/apache/hudi/pull/7307
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1328465658
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "53575f2200794996c14749d78afce6535d5272ac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13265",
"triggerID" : "53575f2200794996c14749d78afce6535d5272ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53575f2200794996c14749d78afce6535d5272ac Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13265)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by GitBox <gi...@apache.org>.
TengHuo commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1362316237
Hi
is there anyone can help to review it? Really appreciate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1413220811
Agree @danny0405, think it's better we unify Avro schema handling across Spark and Flink in Hudi.
Currently, we have Avro schema tools class `org.apache.hudi.avro.AvroSchemaUtils` in module `hudi-common` to manipulate Avro schema. And Hudi Spark is using `org.apache.spark.sql.avro.SchemaConverters` to do conversion between Spark DataType and Avro schema. Hudi Flink is using ` org.apache.hudi.util.AvroSchemaConverter` to do conversion between Flink DataType and Avro schema.
I noticed that there is different behaviour when setting the name of a new Avro schema.
**In Spark side**, it exposes the name and namespace of Avro schema as method parameter.
```java
/**
* Converts a Spark SQL schema to a corresponding Avro schema.
*
* @since 2.4.0
*/
def toAvroType(catalystType: DataType,
nullable: Boolean = false,
recordName: String = "topLevelRecord",
nameSpace: String = ""): Schema
```
reference: https://github.com/apache/hudi/blob/41653fc708854828bacb23ed624ca6b3a67d6737/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala#L154
**In Flink side**, it uses a constant name `"record"`
```java
/**
* Converts Flink SQL {@link LogicalType} (can be nested) into an Avro schema.
*
* <p>Use "record" as the type name.
*
* @param schema the schema type, usually it should be the top level record type, e.g. not a
* nested type
* @return Avro's {@link Schema} matching this logical type.
*/
public static Schema convertToSchema(LogicalType schema) {
return convertToSchema(schema, "record");
}
```
reference: https://github.com/apache/hudi/blob/8ffcb2fc9470077bdcf3810756545d081fb6523c/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/AvroSchemaConverter.java#L202
(please correct me if I'm wrong)
May I know if it is possible we unify all non engine related schemas things in one place? e.g. name conversion rule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1403222206
> Hi @alexeykudinkin
>
> Before rebase, I have one thing want to check with you.
>
> Last week, there was an issue about a similar exception about Avro schema namespace, #7691.
>
> And @danny0405 mentioned in that ticket that it uses a constant namespace "record" in Flink side, [#7691 (comment)](https://github.com/apache/hudi/issues/7691#issuecomment-1386486937).
>
> And in Spark side, I found we are using a namespace pattern `"namespace": "hoodie.test_mor_tab"` (`test_mor_tab` is Hudi table name) in writer schema, and a constant `"name": "Record"` in reader schema. [#7284 (comment)](https://github.com/apache/hudi/issues/7284#issuecomment-1324899843)
>
> May I ask which one we should follow? Think we need to keep it consistent between Spark and Flink.
If the namespace check is an Avro behavior and there is no way to work around, I'm afraid we must unify all the avro schema name spaces for read/writer schema then, does the `hoodie.table_name` namespace makes any sense here? How about we all use the constant `record` as the namespace name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1328623380
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "53575f2200794996c14749d78afce6535d5272ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13265",
"triggerID" : "53575f2200794996c14749d78afce6535d5272ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53575f2200794996c14749d78afce6535d5272ac Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13265)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1415147727
> parameter
I'm +1 too to unify the avro record schema namespace, let's make the namespace parametric for Flink tool `#AvroSchemaConverter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1403077872
Hi @alexeykudinkin
Before rebase, I have one thing want to check with you.
Last week, there was an issue about a similar exception about Avro schema namespace, https://github.com/apache/hudi/issues/7691.
And @danny0405 mentioned in that ticket that it uses a constant namespace "record" in Flink side, https://github.com/apache/hudi/issues/7691#issuecomment-1386486937.
And in Spark side, I found we are using a namespace pattern `"namespace": "hoodie.test_mor_tab"` (`test_mor_tab` is Hudi table name) in writer schema, and a
constant `"name": "Record"` in reader schema. https://github.com/apache/hudi/issues/7284#issuecomment-1324899843
May I ask which one we should follow? Think we need to keep it consistent between Spark and Flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1413376520
@TengHuo Thanks for the detailed comment. I do agree that we should unify the schema. It bodes well for engine interoperability as well. Does this PR already cover the unification?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1506631972
This issue has been fixed in the latest master. This test case can pass now. Detail in https://github.com/apache/hudi/issues/7284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by GitBox <gi...@apache.org>.
TengHuo commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1328817142
And I found there is another similar issue in `HoodieCatalogTable#initHoodieTable`
https://github.com/apache/hudi/blob/e088faac9eca747f74a48f12358a3bb6f66c21d5/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala#L189
When creating a table from an empty path, it will set the record name in `hoodie.table.create.schema` as `$tableName_record`.
However, if the path is an existing Hudi table (`hoodieTableExists` is `true`) when creating a table, it will rewrite the record name in `hoodie.table.create.schema` as `topLevelRecord`, which is the default value of `recordName` in `SchemaConverters#toAvroType`.
May I ask if it is an issue or by design behaviour?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1403059007
> @TengHuo can you please rebase to the latest master and verify whether fix is still relevant?
@alexeykudinkin got it, np. Let me rebase it to the latest master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #7307: [HUDI-5271] fix issue inconsistent reader and writer schema in HoodieAvroDataBlock
Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #7307:
URL: https://github.com/apache/hudi/pull/7307#issuecomment-1413391784
> @TengHuo Thanks for the detailed comment. I do agree that we should unify the schema. It bodes well for engine interoperability as well. Does this PR already cover the unification? @danny0405 What do you think?
@codope
nope, this PR doesn't include schema unification. I was planning to fix an Avro schema namespace inconsistent issue in Spark side only. Then I notice there are different Avro schema name & namespace handling code between Flink and Spark.
So I think my fix in this PR can't solve this issue for all situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org