You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "steve-xi-awx (via GitHub)" <gi...@apache.org> on 2024/03/06 09:38:05 UTC
[I] The BigQuerySyncTool can't work well when the hudi table schema changed [SUPPORT] [hudi]
steve-xi-awx opened a new issue, #10829:
URL: https://github.com/apache/hudi/issues/10829
**To Reproduce**
Steps to reproduce the behavior:
1. use BigQuerySyncTool to sync hudi table into BQ as a external table with connection id
2. update the hudi table schema, then when we try to sync the hudi table metadata to BigQuery table, the error is occurred.
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
error when sync to BigQuery, will ignore this error and continue to process next batch, error is An error occurred while calling o2983.syncHoodieTable.
: com.google.cloud.bigquery.BigQueryException: Schema can be specified only on the Table.Schema field for external tables with an associated connection_id but schema was provided on Table.Externaldataconfig.Schema.
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.patch(HttpBigQueryRpc.java:271)
at com.google.cloud.bigquery.BigQueryImpl$15.call(BigQueryImpl.java:673)
at com.google.cloud.bigquery.BigQueryImpl$15.call(BigQueryImpl.java:670)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.update(BigQueryImpl.java:669)
at org.apache.hudi.gcp.bigquery.HoodieBigQuerySyncClient.updateTableSchema(HoodieBigQuerySyncClient.java:206)
at org.apache.hudi.gcp.bigquery.BigQuerySyncTool.syncTable(BigQuerySyncTool.java:147)
at org.apache.hudi.gcp.bigquery.BigQuerySyncTool.syncHoodieTable(BigQuerySyncTool.java:111)
at sun.reflect.GeneratedMethodAccessor817.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.sendCommand(ClientServerConnection.java:244)
at py4j.CallbackClient.sendCommand(CallbackClient.java:384)
at py4j.CallbackClient.sendCommand(CallbackClient.java:356)
at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:106)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The BigQuerySyncTool can't work well when the hudi table schema changed [SUPPORT] [hudi]
Posted by "steve-xi-awx (via GitHub)" <gi...@apache.org>.
steve-xi-awx commented on issue #10829:
URL: https://github.com/apache/hudi/issues/10829#issuecomment-1982315857
I have raised a mr for this issue and it seems the change can fix that issue.
https://github.com/apache/hudi/pull/10830
I think this problem is caused by that the external table in BigQuery with a connection id should specify the table schema in the wrong position.
Your sample did't specify the connection id so that the table is still a simple external table, not a Big Lake table. This problem is occurred in release-0.14.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The BigQuerySyncTool can't work well when the hudi table schema changed [SUPPORT] [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10829:
URL: https://github.com/apache/hudi/issues/10829#issuecomment-1982325180
> @ad1happy2go Can you help review this mr ?
I will take it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The BigQuerySyncTool can't work well when the hudi table schema changed [SUPPORT] [hudi]
Posted by "steve-xi-awx (via GitHub)" <gi...@apache.org>.
steve-xi-awx commented on issue #10829:
URL: https://github.com/apache/hudi/issues/10829#issuecomment-1982316248
@ad1happy2go Can you help review this mr ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The BigQuerySyncTool can't work well when the hudi table schema changed [SUPPORT] [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10829:
URL: https://github.com/apache/hudi/issues/10829#issuecomment-1980827448
@steve-xi-awx Thanks a lot for raising this. What kind of schema change is happening? Can you post the writer configuration and big query sync configuration? I tried to add a new column, and it ran without any exception. Can you check code and let me know in case I am missing anything. You can also take this code and then try to reproduce with sample dataset.
https://gist.github.com/ad1happy2go/17b32db63f68b49813c8430967a99ec8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The BigQuerySyncTool can't work well when the hudi table schema changed [SUPPORT] [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10829:
URL: https://github.com/apache/hudi/issues/10829#issuecomment-1982360349
Thanks @steve-xi-awx for the fix. Thanks @danny0405 .
Tracking JIRA - https://issues.apache.org/jira/browse/HUDI-7488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The BigQuerySyncTool can't work well when the hudi table schema changed [SUPPORT] [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #10829: The BigQuerySyncTool can't work well when the hudi table schema changed [SUPPORT]
URL: https://github.com/apache/hudi/issues/10829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org