You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xuzifu666 (via GitHub)" <gi...@apache.org> on 2024/03/05 09:07:29 UTC
[I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
xuzifu666 opened a new issue, #10819:
URL: https://github.com/apache/hudi/issues/10819
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1. Data write to Hudi table and mdt is open;
2.(set hoodie.metadata.enable=true;
set hoodie.enable.data.skipping=true;
set hoodie.metadata.index.column.stats.enable=true;)
3. Query data with cols filter
4. Data query is slow as before use mdt
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.14.0
* Spark version : 3.2.0
* Hive version : 1.1.0
* Hadoop version : 3.2
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978280763
@ad1happy2go could you look into this issue? thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978678805
@xuzifu666 Why do you think this config is not working. Were you able to locate that log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978394912
@xuzifu666 Looks like data skipping is happening.
We are logging below. Can you locate below log in your driver logs and compare both ro and rt table.
```
logInfo(s"Total file slices: $totalFileSliceSize; " +
s"candidate file slices after data skipping: $candidateFileSliceSize; " +
s"skipping percentage $skippingRatio")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978284964
@xuzifu666 Will look into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978345857
If sql is select * from table where col = 'v1' limit 10;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978294901
> @xuzifu666 Will look into it. So querying rt with data skipping or without data skipping taking same amount of time.
Yes，seem not work well with data skipping，need to confirm the config is right which is according to the doc and source code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978296721
@xuzifu666 Config looks okay. Can you share spark UI and compare both the runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978685291
> @xuzifu666 Why do you think this config is not working. Were you able to locate that log?
>
it is sparkSQL reason not relate to hudi，I would add resolve method to the issue latter·thanks@ad1happy2go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978398785
> skipping percentage
data skipping should happen，but hoodie.datasource.read.file.index.listing.mode seems not work，list file slice many times with increase num 1、4、20、100 @ad1happy2go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Query with MDT but time cost is much as not on mdt [hudi]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on issue #10819:
URL: https://github.com/apache/hudi/issues/10819#issuecomment-1978311324
> @xuzifu666 Config looks okay. Can you share spark UI and compare both the runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org