You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/17 19:49:16 UTC
[GitHub] [hudi] sambhav13 opened a new issue, #6125: [SUPPORT] hudi-examples-dbt not running with spark thrift server
sambhav13 opened a new issue, #6125:
URL: https://github.com/apache/hudi/issues/6125
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Tried Running https://github.com/apache/hudi/tree/master/hudi-examples/hudi-examples-dbt . Started the spark Thrift server on 10000 port and configured that in dbt. On running dbt run , error comes saying **org.apache.hudi.exception.HoodieException: 'hoodie.table.name' must be set.**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1. Start thrift server locally using ./sbin/start-thriftserver.sh --jars hudi-spark-bundle_2.11-0.10.1.jar using spark-3.1.1-bin-hadoop3.2
2. Set up ~/.dbt/profiles.yaml
learnDbt:
outputs:
dev:
host: localhost
method: thrift
port: 10000
schema: sales
threads: 1
type: spark
target: dev
spark:
outputs:
dev:
host: localhost
method: thrift
port: 10000
schema: sales
threads: 1
type: spark
target: dev
3. dbt debug to test connectivity
4. dbt run to create models.
**Expected behavior**
As expected on the dbt run command the command should be able to successfully create models.
**Environment Description**
local cloned repo on macOs.
* Hudi version :
0.11
* Spark version :
3.1.1
* Hive version :
* Hadoop version :
* 3.2
* Storage (HDFS/S3/GCS..) :
Local Storage
* Running on Docker? (yes/no) :
no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
" Database Error
10:38:28 org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.hudi.exception.HoodieException: 'hoodie.table.name' must be set"
```Add the stacktrace of the error.```
<img width="1904" alt="dbt-run-with_hudi-examples-dbt" src="https://user-images.githubusercontent.com/4696334/179422321-84f9514b-3ee6-4988-924b-2aff80f5d670.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6125: [SUPPORT] hudi-examples-dbt not running with spark thrift server
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6125:
URL: https://github.com/apache/hudi/issues/6125#issuecomment-1209902082
@vingov : can u assist here please.
@minihippo : if you have time, can you follow up as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hudi-examples-dbt not running with spark thrift server [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan closed issue #6125: [SUPPORT] hudi-examples-dbt not running with spark thrift server
URL: https://github.com/apache/hudi/issues/6125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sambhav13 commented on issue #6125: [SUPPORT] hudi-examples-dbt not running with spark thrift server
Posted by GitBox <gi...@apache.org>.
sambhav13 commented on issue #6125:
URL: https://github.com/apache/hudi/issues/6125#issuecomment-1187599344
@minihippo Can you elaborate a bit what do you mean by spark-submit/sql? What I did was just clone the hudi repo and tried with dbt with spark thrift server configured as backend engine for dbt to run the hudi-examples-dbt models. I think it should not matter what way spark is being run , It could be due to some property missing but I am relying on the hudi-examples-dbt to have all that already .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on issue #6125: [SUPPORT] hudi-examples-dbt not running with spark thrift server
Posted by GitBox <gi...@apache.org>.
minihippo commented on issue #6125:
URL: https://github.com/apache/hudi/issues/6125#issuecomment-1187350017
@sambhav13 Can u provide the failed sql? BTW, currently the spark writer design is conflict with spark thrift server, especially we have the kill/cancel operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hudi-examples-dbt not running with spark thrift server [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #6125:
URL: https://github.com/apache/hudi/issues/6125#issuecomment-1815182853
@sambhav13 We confirmed it works well with spark thrift server. Tried hudi 0.12.3 , 0.13.1 and 0.14.0 and all works good.
Sorry for the delay in triaging this. Do you still face this issue. I can help if you still not able to make it run. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hudi-examples-dbt not running with spark thrift server [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on issue #6125:
URL: https://github.com/apache/hudi/issues/6125#issuecomment-1828246693
closing as solution provided
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #6125: [SUPPORT] hudi-examples-dbt not running with spark thrift server
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #6125:
URL: https://github.com/apache/hudi/issues/6125#issuecomment-1186639295
hey @vingov can you take a look please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hudi-examples-dbt not running with spark thrift server [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on issue #6125:
URL: https://github.com/apache/hudi/issues/6125#issuecomment-1822221787
@sambhav13 I'm updating the instructions in the dbt example (using spark 3.2 and hudi 0.14.0). Please check this out and let us know if it helps.
https://github.com/apache/hudi/blob/af912a3fed7270708fad935b7df55fb508cd5536/hudi-examples/hudi-examples-dbt/README.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org