You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Ssv-21 (via GitHub)" <gi...@apache.org> on 2025/01/20 02:06:04 UTC
[I] [SUPPORT] Does AWS Redshift Spectrum fully support Hudi meta table features like column stats and bloom filters? [hudi]
Ssv-21 opened a new issue, #12674:
URL: https://github.com/apache/hudi/issues/12674
I’m trying to understand how well AWS Redshift Spectrum supports data skipping when working with Hudi datasets.
From the [Athena documentation](https://docs.aws.amazon.com/athena/latest/ug/querying-hudi-metadata-table.html), it’s clear that Athena supports file listing from Hudi’s meta table but doesn’t utilize features like column statistics or bloom filters for data skipping. However, the official docs do not mention this for Redshift Spectrum.
Does anyone know if Redshift Spectrum, Supports column statistics and bloom filters from Hudi’s meta table for query optimization?
or it Works the same as Athena by only supporting file listing?
Any clarification or references would be really helpful. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Does AWS Redshift Spectrum fully support Hudi meta table features like column stats and bloom filters? [hudi]
Posted by "Ssv-21 (via GitHub)" <gi...@apache.org>.
Ssv-21 commented on issue #12674:
URL: https://github.com/apache/hudi/issues/12674#issuecomment-2692801613
Hi @rangareddy,
I appreciate your reply, My question is specifically about Redshift Spectrum's data-skipping capabilities. Does it leverage column statistics or bloom filters generated and embedded in Hudi tables during writing?
Any references around this would be really helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Does AWS Redshift Spectrum fully support Hudi meta table features like column stats and bloom filters? [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #12674:
URL: https://github.com/apache/hudi/issues/12674#issuecomment-2602172282
Hi @Ssv-21
The following documentation for Redshift Spectrum has been located. However, it does not provide information regarding the feature's support status. We will therefore need to contact the AWS product team for further clarification.
- https://docs.aws.amazon.com/redshift/latest/dg/c-spectrum-external-tables.html
From onehouse blog,
"Hudi supports snapshot isolation, which means you can query data without picking up any in-progress or not-yet-committed changes. This means you can write changes to the dataset in EMR at the same time users are querying it with Athena and Amazon Redshift Spectrum."
- https://www.onehouse.ai/blog/apache-hudi-native-aws-integrations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Does AWS Redshift Spectrum fully support Hudi meta table features like column stats and bloom filters? [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #12674:
URL: https://github.com/apache/hudi/issues/12674#issuecomment-2674525822
Hi @Ssv-21
Please let me know if the above information will help you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Does AWS Redshift Spectrum fully support Hudi meta table features like column stats and bloom filters? [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #12674:
URL: https://github.com/apache/hudi/issues/12674#issuecomment-2705867070
@ad1happy2go Could you please help me identify the contact person from the AWS side who can provide an answer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org