You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "masthanmca (via GitHub)" <gi...@apache.org> on 2024/02/06 12:27:34 UTC
[I] [SUPPORT] BQ synch tool not working with HUDI bundle jar [hudi]
masthanmca opened a new issue, #10629:
URL: https://github.com/apache/hudi/issues/10629
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? yes
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
BQ sync is not working with hudi bundle jar
A clear and concise description of the problem.
I wanted to enable BQ sync while writing ingest the data into HUDI table using manifest file.
**To Reproduce**
Steps to reproduce the behavior:
1. create data frame with any schema
2. use the below options for Bq sync along with the other default HUDI configurations
3. hiveConfigs.put("org.apache.hudi.gcp.bigquery.BigQuerySyncTool", "true")
hiveConfigs.put("hoodie.gcp.bigquery.sync.project_id", bqSyncProjectId)
hiveConfigs.put("hoodie.gcp.bigquery.sync.dataset_name", bqSyncDatasetName)
hiveConfigs.put("hoodie.gcp.bigquery.sync.table_name", hoodieHiveSyncTable)
hiveConfigs.put("hoodie.gcp.bigquery.sync.dataset_location", "us")
hiveConfigs.put("hoodie.gcp.bigquery.sync.source_uri", bqSyncSourceUri)
hiveConfigs.put("hoodie.gcp.bigquery.sync.source_uri_prefix", bqSyncSourceUriPrefix)
hiveConfigs.put("hoodie.gcp.bigquery.sync.base_path", bqSyncBasePath)
hiveConfigs.put("hoodie.gcp.bigquery.sync.partition_fields", hoodieHiveSyncPartitionFields)
hiveConfigs.put("hoodie.gcp.bigquery.sync.use_bq_manifest_file", "true")
4. write the data frame in HUDI table.ds.write.format(HudiFormat).options(hoodieConfigs).options(hiveConfigs).mode(writeMode).save(location)
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.14.0
* Spark version : 3.3.2
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : GCS
* Running on Docker? (yes/no) :no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
No error , but external table not created in Big Query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] BQ synch tool not working with HUDI bundle jar [hudi]
Posted by "abhishekshenoy (via GitHub)" <gi...@apache.org>.
abhishekshenoy commented on issue #10629:
URL: https://github.com/apache/hudi/issues/10629#issuecomment-1951685995
Facing the same issue , does not work with hudi-spark3.3-bundle jar .
Hudi Write to path works , Hive Sync works but BQ sync does not work.
For now have taken this route based on a flag to manually perform the BQSync with BQSyncTool post the dataframe.write
https://github.com/apache/hudi/issues/9355#issuecomment-1696764242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] BQ synch tool not working with HUDI bundle jar [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10629:
URL: https://github.com/apache/hudi/issues/10629#issuecomment-1952307824
@abhishekshenoy @masthanmca That (https://github.com/apache/hudi/issues/9355#issuecomment-1696764242) i.e. BigQuerySyncTool is the correct way of doing BQ sync with batch jobs.
The another way is doing this with HudiStreamer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] BQ synch tool not working with HUDI bundle jar [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #10629: [SUPPORT] BQ synch tool not working with HUDI bundle jar
URL: https://github.com/apache/hudi/issues/10629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] BQ synch tool not working with HUDI bundle jar [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10629:
URL: https://github.com/apache/hudi/issues/10629#issuecomment-1958947812
@masthanmca @abhishekshenoy I went through the code and identified that we need to set both the class names to do both metastync together. The default value for below prop is just hive sync. I tried with 0.14.1 hudi version and after write and hive sync completed, it tried to do Big query sync also.
```
"hoodie.meta.sync.client.tool.class" : "org.apache.hudi.hive.HiveSyncTool,org.apache.hudi.gcp.bigquery.BigQuerySyncTool"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] BQ synch tool not working with HUDI bundle jar [hudi]
Posted by "abhishekshenoy (via GitHub)" <gi...@apache.org>.
abhishekshenoy commented on issue #10629:
URL: https://github.com/apache/hudi/issues/10629#issuecomment-1953462323
@ad1happy2go @the-other-tim-brown
```
But should nt that be internally called when we are providing the Hudi Bq configs and enabling META_SYNC_ENABLED.
In my case we use df.write.options(hudiAndHiveAndBQConfigs).save() and the hudiAndHiveAndBQConfigs has both hive and bq related configs . **But still only hive sync happens implicitly**.
Is it by design that as part of our write function we need to perform both
df.write.options(hudiAndHiveAndBQConfigs).save()
new BigQuerySyncTool(getBigQueryProps).syncHoodieTable()
```
```
**Also i am seeing an issue that with BQ sync , the partition fields do not get reflected correctly .**
Our output looks something like this
gs://bucket/folder/tablename/partition_column=partition_value_1
gs://bucket/folder/tablename/partition_column=partition_value_2
We have given the below and the table in BQ is not partitioned
hoodie.gcp.bigquery.sync.base_path=gs://bucket/folder/tablename
hoodie.datasource.meta.sync.base.path=gs://bucket/folder/tablename
hoodie.gcp.bigquery.sync.source_uri_prefix=gs://bucket/folder/tablename
hoodie.datasource.hive_sync.partition_fields=partition_column
hoodie.datasource.hive_sync.partition_fields=partition_column
hoodie.gcp.bigquery.sync.partition_fields=partition_column
hoodie.gcp.bigquery.sync.assume_date_partitioning=false
hoodie.gcp.bigquery.sync.require_partition_filter=false
```
```
If we pass sourceUriPrefix as any of the below , sync fails hoodie.gcp.bigquery.sync.source_uri_prefix=gs://bucket/folder/tablename/partion_column=
hoodie.gcp.bigquery.sync.source_uri_prefix=gs://bucket/folder/tablename/partion_column= *
How can this be resolved
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] BQ synch tool not working with HUDI bundle jar [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10629:
URL: https://github.com/apache/hudi/issues/10629#issuecomment-1966839409
@masthanmca Closing out this issue as I confirmed it works. Please reopen in case you still see this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] BQ synch tool not working with HUDI bundle jar [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10629:
URL: https://github.com/apache/hudi/issues/10629#issuecomment-1929778516
@masthanmca
Is the the first time you are facing this issue or after upgrade you started facing this one.
Your configurations also looks wrong? From where you got these or which doc you referred?
can you refer - https://hudi.apache.org/docs/gcp_bigquery/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org