You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/12/05 21:10:27 UTC
[PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
the-other-tim-brown opened a new pull request, #17498:
URL: https://github.com/apache/hudi/pull/17498
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
Migrates the Avro to BigQuery Schema converter to use HoodieSchema as part of our migration to our own type system.
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
- Updates the converter to operate on the HoodieSchema
- Updates test objects to avoid usage of Avro
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
Continues migration towards our own type system
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
Low, this is covered pretty extensively by unit testing
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3618619333
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17498:
URL: https://github.com/apache/hudi/pull/17498#discussion_r2616440914
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySchemaResolver.java:
##########
@@ -100,34 +100,34 @@ private static String fieldsToSqlString(List<Field> fields) {
}
@VisibleForTesting
- Schema convertSchema(org.apache.avro.Schema schema) {
+ Schema convertSchema(HoodieSchema schema) {
return Schema.of(getFields(schema));
}
- private Field getField(org.apache.avro.Schema fieldSchema, String name, boolean nullable) {
+ private Field getField(HoodieSchema fieldSchema, String name, boolean nullable) {
final Field.Mode fieldMode = nullable ? Field.Mode.NULLABLE : Field.Mode.REQUIRED;
StandardSQLTypeName standardSQLTypeName;
switch (fieldSchema.getType()) {
case INT:
case LONG:
- LogicalType logicalType = fieldSchema.getLogicalType();
- if (logicalType == null) {
- standardSQLTypeName = StandardSQLTypeName.INT64;
- } else if (logicalType.equals(LogicalTypes.date())) {
- standardSQLTypeName = StandardSQLTypeName.DATE;
- } else if (logicalType.equals(LogicalTypes.timeMillis()) || logicalType.equals(LogicalTypes.timeMicros())) {
- standardSQLTypeName = StandardSQLTypeName.TIME;
- } else if (logicalType.equals(LogicalTypes.timestampMillis()) || logicalType.equals(LogicalTypes.timestampMicros())) {
+ standardSQLTypeName = StandardSQLTypeName.INT64;
+ break;
+ case TIME:
+ standardSQLTypeName = StandardSQLTypeName.TIME;
+ break;
+ case TIMESTAMP:
+ HoodieSchema.Timestamp timestampType = (HoodieSchema.Timestamp) fieldSchema;
+ if (timestampType.isUtcAdjusted()) {
standardSQLTypeName = StandardSQLTypeName.TIMESTAMP;
- // Due to older avro support, we need to use strings for local timestamp logical types
- } else if (logicalType.getName().equals("local-timestamp-millis") || logicalType.getName().equals("local-timestamp-micros")) {
- standardSQLTypeName = StandardSQLTypeName.INT64;
} else {
- throw new IllegalArgumentException("Unexpected logical type in schema: " + logicalType);
+ standardSQLTypeName = StandardSQLTypeName.INT64;
}
break;
- case ENUM:
+ case DATE:
+ standardSQLTypeName = StandardSQLTypeName.DATE;
+ break;
case STRING:
+ case ENUM:
Review Comment:
Should `UUID` type be handled as `STRING`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3649732126
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17498:
URL: https://github.com/apache/hudi/pull/17498#discussion_r2616536471
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySchemaResolver.java:
##########
@@ -100,34 +100,34 @@ private static String fieldsToSqlString(List<Field> fields) {
}
@VisibleForTesting
- Schema convertSchema(org.apache.avro.Schema schema) {
+ Schema convertSchema(HoodieSchema schema) {
return Schema.of(getFields(schema));
}
- private Field getField(org.apache.avro.Schema fieldSchema, String name, boolean nullable) {
+ private Field getField(HoodieSchema fieldSchema, String name, boolean nullable) {
final Field.Mode fieldMode = nullable ? Field.Mode.NULLABLE : Field.Mode.REQUIRED;
StandardSQLTypeName standardSQLTypeName;
switch (fieldSchema.getType()) {
case INT:
case LONG:
- LogicalType logicalType = fieldSchema.getLogicalType();
- if (logicalType == null) {
- standardSQLTypeName = StandardSQLTypeName.INT64;
- } else if (logicalType.equals(LogicalTypes.date())) {
- standardSQLTypeName = StandardSQLTypeName.DATE;
- } else if (logicalType.equals(LogicalTypes.timeMillis()) || logicalType.equals(LogicalTypes.timeMicros())) {
- standardSQLTypeName = StandardSQLTypeName.TIME;
- } else if (logicalType.equals(LogicalTypes.timestampMillis()) || logicalType.equals(LogicalTypes.timestampMicros())) {
+ standardSQLTypeName = StandardSQLTypeName.INT64;
+ break;
+ case TIME:
+ standardSQLTypeName = StandardSQLTypeName.TIME;
+ break;
+ case TIMESTAMP:
+ HoodieSchema.Timestamp timestampType = (HoodieSchema.Timestamp) fieldSchema;
+ if (timestampType.isUtcAdjusted()) {
standardSQLTypeName = StandardSQLTypeName.TIMESTAMP;
- // Due to older avro support, we need to use strings for local timestamp logical types
- } else if (logicalType.getName().equals("local-timestamp-millis") || logicalType.getName().equals("local-timestamp-micros")) {
- standardSQLTypeName = StandardSQLTypeName.INT64;
} else {
- throw new IllegalArgumentException("Unexpected logical type in schema: " + logicalType);
+ standardSQLTypeName = StandardSQLTypeName.INT64;
}
break;
- case ENUM:
+ case DATE:
+ standardSQLTypeName = StandardSQLTypeName.DATE;
+ break;
case STRING:
+ case ENUM:
Review Comment:
Yes, good catch. I am updating the PR to cover this in a test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3649664796
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17498:
URL: https://github.com/apache/hudi/pull/17498#discussion_r2608850962
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySchemaResolver.java:
##########
@@ -100,34 +100,34 @@ private static String fieldsToSqlString(List<Field> fields) {
}
@VisibleForTesting
- Schema convertSchema(org.apache.avro.Schema schema) {
+ Schema convertSchema(HoodieSchema schema) {
return Schema.of(getFields(schema));
}
- private Field getField(org.apache.avro.Schema fieldSchema, String name, boolean nullable) {
+ private Field getField(HoodieSchema fieldSchema, String name, boolean nullable) {
final Field.Mode fieldMode = nullable ? Field.Mode.NULLABLE : Field.Mode.REQUIRED;
StandardSQLTypeName standardSQLTypeName;
switch (fieldSchema.getType()) {
case INT:
case LONG:
- LogicalType logicalType = fieldSchema.getLogicalType();
- if (logicalType == null) {
- standardSQLTypeName = StandardSQLTypeName.INT64;
- } else if (logicalType.equals(LogicalTypes.date())) {
- standardSQLTypeName = StandardSQLTypeName.DATE;
- } else if (logicalType.equals(LogicalTypes.timeMillis()) || logicalType.equals(LogicalTypes.timeMicros())) {
- standardSQLTypeName = StandardSQLTypeName.TIME;
- } else if (logicalType.equals(LogicalTypes.timestampMillis()) || logicalType.equals(LogicalTypes.timestampMicros())) {
+ standardSQLTypeName = StandardSQLTypeName.INT64;
+ break;
+ case TIME:
+ standardSQLTypeName = StandardSQLTypeName.TIME;
+ break;
+ case TIMESTAMP:
+ HoodieSchema.Timestamp timestampType = (HoodieSchema.Timestamp) fieldSchema;
+ if (timestampType.isUtcAdjusted()) {
standardSQLTypeName = StandardSQLTypeName.TIMESTAMP;
- // Due to older avro support, we need to use strings for local timestamp logical types
- } else if (logicalType.getName().equals("local-timestamp-millis") || logicalType.getName().equals("local-timestamp-micros")) {
Review Comment:
Right, the `local` timestamps are the timestamaps where `isUtcAdjusted` is false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3649824157
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
}, {
"hash" : "51dd282200f103f7116c6da1440ec96727b2a375",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "51dd282200f103f7116c6da1440ec96727b2a375",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958)
* 51dd282200f103f7116c6da1440ec96727b2a375 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3649715016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3649662656
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3618632123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #17498:
URL: https://github.com/apache/hudi/pull/17498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3649663712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3618625043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3639220862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3649904289
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
}, {
"hash" : "51dd282200f103f7116c6da1440ec96727b2a375",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10215",
"triggerID" : "51dd282200f103f7116c6da1440ec96727b2a375",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51dd282200f103f7116c6da1440ec96727b2a375 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10215)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3638206450
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3639209486
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "balaji-varadarajan-ai (via GitHub)" <gi...@apache.org>.
balaji-varadarajan-ai commented on code in PR #17498:
URL: https://github.com/apache/hudi/pull/17498#discussion_r2608792183
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySchemaResolver.java:
##########
@@ -100,34 +100,34 @@ private static String fieldsToSqlString(List<Field> fields) {
}
@VisibleForTesting
- Schema convertSchema(org.apache.avro.Schema schema) {
+ Schema convertSchema(HoodieSchema schema) {
return Schema.of(getFields(schema));
}
- private Field getField(org.apache.avro.Schema fieldSchema, String name, boolean nullable) {
+ private Field getField(HoodieSchema fieldSchema, String name, boolean nullable) {
final Field.Mode fieldMode = nullable ? Field.Mode.NULLABLE : Field.Mode.REQUIRED;
StandardSQLTypeName standardSQLTypeName;
switch (fieldSchema.getType()) {
case INT:
case LONG:
- LogicalType logicalType = fieldSchema.getLogicalType();
- if (logicalType == null) {
- standardSQLTypeName = StandardSQLTypeName.INT64;
- } else if (logicalType.equals(LogicalTypes.date())) {
- standardSQLTypeName = StandardSQLTypeName.DATE;
- } else if (logicalType.equals(LogicalTypes.timeMillis()) || logicalType.equals(LogicalTypes.timeMicros())) {
- standardSQLTypeName = StandardSQLTypeName.TIME;
- } else if (logicalType.equals(LogicalTypes.timestampMillis()) || logicalType.equals(LogicalTypes.timestampMicros())) {
+ standardSQLTypeName = StandardSQLTypeName.INT64;
+ break;
+ case TIME:
+ standardSQLTypeName = StandardSQLTypeName.TIME;
+ break;
+ case TIMESTAMP:
+ HoodieSchema.Timestamp timestampType = (HoodieSchema.Timestamp) fieldSchema;
+ if (timestampType.isUtcAdjusted()) {
standardSQLTypeName = StandardSQLTypeName.TIMESTAMP;
- // Due to older avro support, we need to use strings for local timestamp logical types
- } else if (logicalType.getName().equals("local-timestamp-millis") || logicalType.getName().equals("local-timestamp-micros")) {
Review Comment:
Just checking : This is consistent with the support for timestamp in Hoodie schema. Right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate BigQuery schema converter to use HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17498:
URL: https://github.com/apache/hudi/pull/17498#issuecomment-3649826657
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"triggerType" : "PUSH"
}, {
"hash" : "9c92c4b6b044e9a203d4a42dca144c4e89f0e14d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958",
"triggerID" : "3638206450",
"triggerType" : "MANUAL"
}, {
"hash" : "51dd282200f103f7116c6da1440ec96727b2a375",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10215",
"triggerID" : "51dd282200f103f7116c6da1440ec96727b2a375",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c92c4b6b044e9a203d4a42dca144c4e89f0e14d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10207) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9958)
* 51dd282200f103f7116c6da1440ec96727b2a375 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10215)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org