You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (JIRA)" <ji...@apache.org> on 2019/03/25 17:00:03 UTC
[jira] [Created] (IMPALA-8344) Add support for running S3 tests
with S3Guard
Joe McDonnell created IMPALA-8344:
-------------------------------------
Summary: Add support for running S3 tests with S3Guard
Key: IMPALA-8344
URL: https://issues.apache.org/jira/browse/IMPALA-8344
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 3.3.0
Reporter: Joe McDonnell
Assignee: Joe McDonnell
Impala s3 tests can encounter failures due to s3's consistency model. S3Guard adds consistency to s3 operations to avoid these types of failures:
[https://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/]
Adding support to run tests with S3Guard provides a way to limit flakiness, and it provides coverage for users that would want to use Impala on S3 with S3Guard.
Support will involve adding the appropriate configuration to core-site.xml. In order to maintain the S3Guard index appropriately, file modifications should go through HDFS commands rather than boto s3 commands. Finally, to reduce costs, Impala may want to have a script to purge S3Guard's dynamodb.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org