You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2023/08/21 02:30:26 UTC
[GitHub] [hudi] the-other-tim-brown opened a new pull request, #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
the-other-tim-brown opened a new pull request, #9488:
URL: https://github.com/apache/hudi/pull/9488
### Change Logs
Adds support for syncing the base files of an MoR table to BigQuery so users to can query the table in a "Read Optimized" way like they can do with other query engines.
### Impact
Allows users to query base files of MoR tables from BigQuery.
### Risk level (write none, low medium or high below)
None
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on a diff in pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "rmahindra123 (via GitHub)" <gi...@apache.org>.
rmahindra123 commented on code in PR #9488:
URL: https://github.com/apache/hudi/pull/9488#discussion_r1300458207
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java:
##########
@@ -128,6 +128,12 @@ public class BigQuerySyncConfig extends HoodieSyncConfig implements Serializable
.withDocumentation("Assume standard yyyy/mm/dd partitioning, this"
+ " exists to support backward compatibility. If you use hoodie 0.3.x, do not set this parameter");
+ public static final ConfigProperty<Boolean> BIGQUERY_SYNC_ALLOW_READ_OPTIMIZED_SYNC = ConfigProperty
+ .key("hoodie.gcp.bigquery.sync.allow_read_optimized_sync")
+ .defaultValue(false)
Review Comment:
yeah i Agree, we should just call it out in the docs. Otherwise lgtm, @codope you can help merge when ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9488:
URL: https://github.com/apache/hudi/pull/9488#issuecomment-1687544152
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2619c32ef7543b508b830ebeae2e6d4c1bae591 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9488:
URL: https://github.com/apache/hudi/pull/9488#issuecomment-1685530329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47cd0ae5da72d7d6e8361cf2af2893690c63a518 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #9488:
URL: https://github.com/apache/hudi/pull/9488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9488:
URL: https://github.com/apache/hudi/pull/9488#issuecomment-1685571905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19382",
"triggerID" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47cd0ae5da72d7d6e8361cf2af2893690c63a518 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19382)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9488:
URL: https://github.com/apache/hudi/pull/9488#issuecomment-1685688789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19382",
"triggerID" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47cd0ae5da72d7d6e8361cf2af2893690c63a518 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19382)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9488:
URL: https://github.com/apache/hudi/pull/9488#issuecomment-1687556282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19394",
"triggerID" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2619c32ef7543b508b830ebeae2e6d4c1bae591 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19394)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9488:
URL: https://github.com/apache/hudi/pull/9488#discussion_r1300434964
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java:
##########
@@ -128,6 +128,12 @@ public class BigQuerySyncConfig extends HoodieSyncConfig implements Serializable
.withDocumentation("Assume standard yyyy/mm/dd partitioning, this"
+ " exists to support backward compatibility. If you use hoodie 0.3.x, do not set this parameter");
+ public static final ConfigProperty<Boolean> BIGQUERY_SYNC_ALLOW_READ_OPTIMIZED_SYNC = ConfigProperty
+ .key("hoodie.gcp.bigquery.sync.allow_read_optimized_sync")
+ .defaultValue(false)
Review Comment:
Let's remove the flag. Don't even need to log a warning. We can call out in the docs that RO queries will be supported for BQ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #9488:
URL: https://github.com/apache/hudi/pull/9488#discussion_r1299540031
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java:
##########
@@ -128,6 +128,12 @@ public class BigQuerySyncConfig extends HoodieSyncConfig implements Serializable
.withDocumentation("Assume standard yyyy/mm/dd partitioning, this"
+ " exists to support backward compatibility. If you use hoodie 0.3.x, do not set this parameter");
+ public static final ConfigProperty<Boolean> BIGQUERY_SYNC_ALLOW_READ_OPTIMIZED_SYNC = ConfigProperty
+ .key("hoodie.gcp.bigquery.sync.allow_read_optimized_sync")
+ .defaultValue(false)
Review Comment:
The main reason why I didn't have it enabled by default was since this is explicitly not allowed in the code today so I wanted to be on the safe side. I don't think there is any plan for Google Cloud to support snapshot query, have you heard otherwise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9488:
URL: https://github.com/apache/hudi/pull/9488#issuecomment-1686850438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19382",
"triggerID" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"triggerType" : "PUSH"
}, {
"hash" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19394",
"triggerID" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47cd0ae5da72d7d6e8361cf2af2893690c63a518 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19382)
* f2619c32ef7543b508b830ebeae2e6d4c1bae591 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19394)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9488:
URL: https://github.com/apache/hudi/pull/9488#discussion_r1299533082
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java:
##########
@@ -128,6 +128,12 @@ public class BigQuerySyncConfig extends HoodieSyncConfig implements Serializable
.withDocumentation("Assume standard yyyy/mm/dd partitioning, this"
+ " exists to support backward compatibility. If you use hoodie 0.3.x, do not set this parameter");
+ public static final ConfigProperty<Boolean> BIGQUERY_SYNC_ALLOW_READ_OPTIMIZED_SYNC = ConfigProperty
+ .key("hoodie.gcp.bigquery.sync.allow_read_optimized_sync")
+ .defaultValue(false)
Review Comment:
Shall we enable by default? For other engines, when we started, we supported COW and MOR read optimized by default.
Also, let's track a ticket to deprecate this config once snapshot query is also supported for MOR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9488:
URL: https://github.com/apache/hudi/pull/9488#issuecomment-1687091730
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19382",
"triggerID" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"triggerType" : "PUSH"
}, {
"hash" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19394",
"triggerID" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2619c32ef7543b508b830ebeae2e6d4c1bae591 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19394)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #9488:
URL: https://github.com/apache/hudi/pull/9488#discussion_r1300367026
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java:
##########
@@ -128,6 +128,12 @@ public class BigQuerySyncConfig extends HoodieSyncConfig implements Serializable
.withDocumentation("Assume standard yyyy/mm/dd partitioning, this"
+ " exists to support backward compatibility. If you use hoodie 0.3.x, do not set this parameter");
+ public static final ConfigProperty<Boolean> BIGQUERY_SYNC_ALLOW_READ_OPTIMIZED_SYNC = ConfigProperty
+ .key("hoodie.gcp.bigquery.sync.allow_read_optimized_sync")
+ .defaultValue(false)
Review Comment:
@codope do we even need this flag or will it be enough to just log some warning and update documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #9488:
URL: https://github.com/apache/hudi/pull/9488#discussion_r1300449186
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java:
##########
@@ -128,6 +128,12 @@ public class BigQuerySyncConfig extends HoodieSyncConfig implements Serializable
.withDocumentation("Assume standard yyyy/mm/dd partitioning, this"
+ " exists to support backward compatibility. If you use hoodie 0.3.x, do not set this parameter");
+ public static final ConfigProperty<Boolean> BIGQUERY_SYNC_ALLOW_READ_OPTIMIZED_SYNC = ConfigProperty
+ .key("hoodie.gcp.bigquery.sync.allow_read_optimized_sync")
+ .defaultValue(false)
Review Comment:
Ok sounds good, I have a todo on my list for this week to go through the docs and make sure they're updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9488: [HUDI-6731] BigQuerySyncTool: add flag to allow for read optimized sync for MoR tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9488:
URL: https://github.com/apache/hudi/pull/9488#issuecomment-1686839177
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19382",
"triggerID" : "47cd0ae5da72d7d6e8361cf2af2893690c63a518",
"triggerType" : "PUSH"
}, {
"hash" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f2619c32ef7543b508b830ebeae2e6d4c1bae591",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47cd0ae5da72d7d6e8361cf2af2893690c63a518 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19382)
* f2619c32ef7543b508b830ebeae2e6d4c1bae591 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org