You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/06 11:09:41 UTC
[GitHub] [hudi] zhangyue19921010 opened a new issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
zhangyue19921010 opened a new issue #3607:
URL: https://github.com/apache/hudi/issues/3607
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
We are testing about presto on hudi with hudi metadata table enable.
But presto server will throw the exception that
```
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.avro.model.HoodieMetadataFileInfo
at org.apache.hudi.metadata.HoodieMetadataPayload.lambda$null$2(HoodieMetadataPayload.java:120)
at java.util.HashMap.forEach(HashMap.java:1289)
at org.apache.hudi.metadata.HoodieMetadataPayload.lambda$createPartitionFilesRecord$3(HoodieMetadataPayload.java:120)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
```
And query is pending forever.
could anyone please take a look and give me a hand?
Thanks in advanced.
**To Reproduce**
Steps to reproduce the behavior:
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.8.0
* presto version : 0.236
* Hive version : 2.3.7
* Hadoop version :xxx
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-927256877
Hi @codope Thanks a lot for your attention. I just tried but failed again :(
Sincerely ask :
1. What's the version of pesto you used?
2. `in docker setup with metadata enabled` => how did you enable metadata table on presto side? I have to modify presto source code to enable presto query using metadata table.
3. Did you just put hudi-presto-bundle into hive-hadoop plugin or did some changing in presto-source pom?
Looking forward to your replay? Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-929111244
> Hi @codope Thanks a lot for your attention. I just tried but failed again :( Sincerely ask :
>
> 1. What's the version of pesto you used?
> 2. `in docker setup with metadata enabled` => how did you enable metadata table on presto side? I have to modify presto source code to enable presto query using metadata table.
> 3. Did you just put hudi-presto-bundle into hive-hadoop plugin or did some changing in presto-source pom?
>
> Looking forward to your replay. Thanks a lot!
1. Docker uses Presto 0.217. But I also built the latest Presto snapshot and tested. However, from Hudi perspective, there should be no change since the Presto version 0.240.
2. I enabled metadata in Hudi, not Presto. Hudi metadata tabled was not synced to Hive.
3. No I did not have to place the bundle or change presto pom. Since Presto 0.240, Hudi is a compile-time dependency already.
Are you able to query normal Hudi data tables using Presto? Are you getting the NoClassDefFoundError only when you query Hudi metadata table? Asking because I did not enable metadata on Presto side. That will require code changes in Presto. Also, youwill need to add hudi-common dependency in Presto pom for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-926467907
@zhangyue19921010 I tried to reproduce this with Hudi 0.8.0 in [docker setup](https://hudi.apache.org/docs/docker_demo) with metadata enabled. Presto query failed but it terminated. However, with the latest master in the same environment, it succeeds and I get correct results. I also verified the metadata on the namenode. Do you want to give latest Hudi a try?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-930110816
@zhangyue19921010 You're right. I think you're talking about #3623
Just left a small comment there. If that works let's merge it. I can also help in testing the dependency changes.
FYI, there is an [open PR](https://github.com/prestodb/presto/pull/15866) for Presto to integrate with Hudi's metadata listing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-926334058
There is a PR out for a long time now in fact. Not Merged
yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-930028288
Are you able to query normal Hudi data tables using Presto? => Yes, normal hudi table works.
Are you getting the NoClassDefFoundError only when you query Hudi metadata table? => Yes
Seems there is dependency conflicts between hudi metadata table(HBase) and presto, I am trying to fix it.
Thanks for your attention :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-926465566
> @nsivabalan Does presto already support Hudi metadata?
@rubenssoto Checkout this PR https://github.com/prestodb/presto/pull/15866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #3607:
URL: https://github.com/apache/hudi/issues/3607
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-930028288
Are you able to query normal Hudi data tables using Presto? => Yes, normal hudi table works.
Are you getting the NoClassDefFoundError only when you query Hudi metadata table? => Yes
Seems there is dependency conflicts between hudi metadata table(HBase) and presto, I am trying to fix it.
Thanks for your attention :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-919730525
@codope : we should be able to try out docker demo w/ presto and metadata table enabled. should be able to quickly get an idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-929111244
> Hi @codope Thanks a lot for your attention. I just tried but failed again :( Sincerely ask :
>
> 1. What's the version of pesto you used?
> 2. `in docker setup with metadata enabled` => how did you enable metadata table on presto side? I have to modify presto source code to enable presto query using metadata table.
> 3. Did you just put hudi-presto-bundle into hive-hadoop plugin or did some changing in presto-source pom?
>
> Looking forward to your replay. Thanks a lot!
1. Docker uses Presto 0.217. But I also built the latest Presto snapshot and tested. However, from Hudi perspective, there should be no change since the Presto version 0.240.
2. I enabled metadata in Hudi, not Presto. Hudi metadata tabled was not synced to Hive.
3. No I did not have to place the bundle or change presto pom. Since Presto 0.240, Hudi is a compile-time dependency already.
Are you able to query normal Hudi data tables using Presto? Are you getting the NoClassDefFoundError only when you query Hudi metadata table? Asking because I did not enable metadata on Presto side. That will require code changes in Presto. Also, youwill need to add hudi-common dependency in Presto pom for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 edited a comment on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-927256877
Hi @codope Thanks a lot for your attention. I just tried but failed again :(
Sincerely ask :
1. What's the version of pesto you used?
2. `in docker setup with metadata enabled` => how did you enable metadata table on presto side? I have to modify presto source code to enable presto query using metadata table.
3. Did you just put hudi-presto-bundle into hive-hadoop plugin or did some changing in presto-source pom?
Looking forward to your replay. Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-930110816
@zhangyue19921010 You're right. I think you're talking about #3623
Just left a small comment there. If that works let's merge it. I can also help in testing the dependency changes.
FYI, there is an [open PR](https://github.com/prestodb/presto/pull/15866) for Presto to integrate with Hudi's metadata listing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-997074101
Since we have patches to support metadata for presto, closing the github issue for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto commented on issue #3607: [SUPPORT]Presto query hudi data with metadata table enable un-successfully.
Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #3607:
URL: https://github.com/apache/hudi/issues/3607#issuecomment-924497696
@nsivabalan Does presto already support Hudi metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org