You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/05/24 01:37:49 UTC
[GitHub] [hudi] jtmzheng opened a new issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?
jtmzheng opened a new issue #2983:
URL: https://github.com/apache/hudi/issues/2983
Is the option https://hudi.apache.org/docs/configurations.html#withConsistencyCheckEnabled still relevant? Given the documentation in https://aws.amazon.com/s3/consistency/ it seems like no ("S3 also provides strong consistency for list operations, so after a write, you can immediately perform a listing of the objects in a bucket with any changes reflected"), but wanted to check before disabling. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?
Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2983:
URL: https://github.com/apache/hudi/issues/2983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2983:
URL: https://github.com/apache/hudi/issues/2983#issuecomment-851717957
@jtmzheng The consistency check is disabled by default
```
private static final String CONSISTENCY_CHECK_ENABLED_PROP = "hoodie.consistency.check.enabled";
private static final String DEFAULT_CONSISTENCY_CHECK_ENABLED = "false";
```
Please confirm with the EMR team to ensure that the S3 flag can be disabled for your use-case. Closing this issue since the config is already false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fanaticjo commented on issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?
Posted by GitBox <gi...@apache.org>.
fanaticjo commented on issue #2983:
URL: https://github.com/apache/hudi/issues/2983#issuecomment-850545801
I think this is still required as i have seen issues still persisting in emr version below 5.33
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtmzheng commented on issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?
Posted by GitBox <gi...@apache.org>.
jtmzheng commented on issue #2983:
URL: https://github.com/apache/hudi/issues/2983#issuecomment-850619877
@fanaticjo what kind of issues did you see?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #2983:
URL: https://github.com/apache/hudi/issues/2983#issuecomment-847412386
I don't think its enabled by default? We should correct the docs, as
needed.
On Mon, May 24, 2021 at 4:07 PM n3nash ***@***.***> wrote:
> @jtmzheng <https://github.com/jtmzheng> That's a good point. I think this
> is not relevant anymore. Do you mind creating a PR to disable this check by
> default as well ?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/hudi/issues/2983#issuecomment-847411435>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAI75PEZ2MXB3DZ45H4OD33TPLL3ZANCNFSM45MJNL3A>
> .
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2983:
URL: https://github.com/apache/hudi/issues/2983#issuecomment-847411435
@jtmzheng That's a good point. I think this is not relevant anymore. Do you mind creating a PR to disable this check by default as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtmzheng commented on issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?
Posted by GitBox <gi...@apache.org>.
jtmzheng commented on issue #2983:
URL: https://github.com/apache/hudi/issues/2983#issuecomment-848289555
We enabled for our Hudi 0.6 dataset since this pre-dated strong consistency on S3. Wanted to confirm we can safely disable this now since I think this adds overhead to ingestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org