You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/10/08 13:26:29 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #3534: HADOOP-17409. Remove s3guard from S3A module
steveloughran opened a new pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534
### Description of PR
* Remove all S3Guard code from S3A production and test modules
* CLI entry point to remove obsolete commands, downgrading to warn and fail
* FS to reject instantiation if metastore set to anything other than null store
### How was this patch tested?
deleting tests until everything worked again (WiP)
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
TODO
- [ ] all tests to complete
- [ ] Remove all tests related to delayed consistency, + a lot of remote file changed
- [ ] remove `whenGuarded()` clauses from cost tests; `withWhenRaw()` to become simply `with()`
- [ ] new CLI entry point "s3a"; "s3guard will map to this"
- [ ] Docs
- [ ] AssumedRole rules & docs
- [ ] Remove the inconsistent client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r783712717
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RenameOperation.java
##########
@@ -724,4 +608,22 @@ private String maybeAddTrailingSlash(String key) {
}
}
+ /**
+ * Convert a passed in exception (expected to be an IOE or AWS exception)
+ * into an IOException.
+ * @param ex exception caught
+ * @return the exception to throw in the failure handler.
+ */
+ protected IOException convertToIOException(final Exception ex) {
+ if (ex instanceof IOException) {
+ return (IOException) ex;
Review comment:
Why casting again?
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -3821,23 +3481,14 @@ S3AFileStatus s3GetFileStatus(final Path path,
// by the time of listing so that the response includes some
// which have not.
- int listSize;
- if (tombstones == null) {
- // no tombstones so look for a marker and at least one child.
- listSize = 2;
- } else {
- // build a listing > tombstones. If the caller has many thousands
- // of tombstones this won't work properly, which is why pruning
- // of expired tombstones matters.
- listSize = Math.min(2 + tombstones.size(), Math.max(2, maxKeys));
- }
+ final int listSize = 2;
S3ListRequest request = createListObjectsRequest(dirKey, "/",
listSize);
// execute the request
S3ListResult listResult = listObjects(request,
getDurationTrackerFactory());
- if (listResult.hasPrefixesOrObjects(contextAccessors, tombstones)) {
+ if (listResult.hasPrefixesOrObjects(contextAccessors, null)) {
Review comment:
Can we remove the tombstones field altogether? I suppose this was only for S3Guard entires.
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -3316,31 +3111,19 @@ protected void maybeCreateFakeParentDirectory(Path path)
}
}
// Here we have a directory which may or may not be empty.
- // So we update the metastore and return.
- return S3Guard.dirListingUnion(
- metadataStore,
- path,
- statusesAssumingNonEmptyDir.getLeft(),
- statusesAssumingNonEmptyDir.getMiddle(),
- allowAuthoritative(path),
- ttlTimeProvider, p ->
- listing.createProvidedFileStatusIterator(
- dirMetaToStatuses(statusesAssumingNonEmptyDir.getMiddle()),
- ACCEPT_ALL,
- Listing.ACCEPT_ALL_BUT_S3N));
+ return statusIt;
}
/**
* Is a path to be considered as authoritative?
- * True iff there is an authoritative metastore or if there
- * is a non-auth store with the supplied path under
+ * is a store with the supplied path under
* one of the paths declared as authoritative.
* @param path path
* @return true if the path is auth
*/
public boolean allowAuthoritative(final Path path) {
return S3Guard.allowAuthoritative(path, this,
- allowAuthoritativeMetadataStore, allowAuthoritativePaths);
+ allowAuthoritativePaths);
Review comment:
Should this be deprecated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011970010
> `[ERROR] Failures: [ERROR] ITestS3AMiscOperations.testEmptyFileChecksums:182->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums of empty files expected:<etag: "5334c7a47521faa37aaaadb3eda3cc7c"> but was:<etag: "37d542d4dade6ea73842e723b57b4c9b"> [ERROR] ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted:258->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums expected:<etag: "057988c6b2a0f57adf8c83c695059978"> but was:<etag: "7ecdd0d908e71fc933749af8612460d7"> [ERROR] Errors: `
>
> Also these failures not sure if they are related.
This is failing in trunk as well even after removing the encryption configs from the auth-keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743662102
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "",
+or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers, for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile, overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB: path names
-and summary details of objects —the actual data is stored in S3, so billed at S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests, combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in `core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0 --write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and `fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
-```
-hadoop s3guard import
-
-import [OPTIONS] [s3a://PATH]
- import metadata from existing S3 data
-
-Common options:
- -authoritative - Mark imported directory data as authoritative.
- -verbose - Verbose Output.
- -meta URL - Metadata repository details (implementation-specific)
-
-Amazon DynamoDB-specific options:
- -region REGION - Service region for connections
-
- URLs for Amazon DynamoDB are of the form dynamodb://TABLE_NAME.
- Specifying both the -region option and an S3A path
- is not supported.
-```
-
-Example
-
-Import all files and directories in a bucket into the S3Guard table.
-
-```bash
-hadoop s3guard import s3a://ireland-1
-```
-
-Import a directory tree, marking directories as authoritative.
-
-```bash
-hadoop s3guard import -authoritative -verbose s3a://ireland-1/fork-0008
-
-2020-01-03 12:05:18,321 [main] INFO - Metadata store DynamoDBMetadataStore{region=eu-west-1,
- tableName=s3guard-metadata, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/s3guard-metadata} is initialized.
-2020-01-03 12:05:18,324 [main] INFO - Starting: Importing s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,324 [main] INFO - Importing directory s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,537 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-false
-2020-01-03 12:05:18,630 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-true
-2020-01-03 12:05:19,142 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-0
-2020-01-03 12:05:19,191 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-1
-2020-01-03 12:05:19,240 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-0
-2020-01-03 12:05:19,289 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-1
-2020-01-03 12:05:19,314 [main] INFO - Updated S3Guard with 0 files and 6 directory entries
-2020-01-03 12:05:19,315 [main] INFO - Marking directory tree s3a://ireland-1/fork-0008 as authoritative
-2020-01-03 12:05:19,342 [main] INFO - Importing s3a://ireland-1/fork-0008: duration 0:01.018s
-Inserted 6 items into Metadata Store
-```
-
-### Compare a S3Guard table and the S3 Store: `s3guard diff`
-
-```bash
-hadoop s3guard diff [-meta URI] s3a://BUCKET
-```
-
-Lists discrepancies between a metadata store and bucket. Note that depending on
-how S3Guard is used, certain discrepancies are to be expected.
-
-Example
-
-```bash
-hadoop s3guard diff s3a://ireland-1
```
Review comment:
did a big review/cull of the doc again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743689411
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "",
+or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers, for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile, overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB: path names
-and summary details of objects —the actual data is stored in S3, so billed at S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests, combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in `core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0 --write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and `fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
Review comment:
cut too much; fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-969227284
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 10s | | trunk passed |
| +1 :green_heart: | compile | 28m 7s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 30s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 23s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 25m 45s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 25m 45s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 22m 18s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 22m 18s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 11s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/results-checkstyle-root.txt) | root: The patch generated 15 new + 71 unchanged - 83 fixed = 86 total (was 154) |
| -1 :x: | mvnsite | 0m 40s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 4s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 43s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 38s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 27m 17s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 51s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 39s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not generate ASF License warnings. |
| | | 247m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle |
| uname | Linux c79218c1d8fe 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2eced9dd71c1ac4bacee97f92e0c63a29db3faae |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011428802
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 109 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 55m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 11s | | trunk passed |
| +1 :green_heart: | compile | 24m 16s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 43s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 51s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 21s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 46s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 54s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 19s | | the patch passed |
| +1 :green_heart: | compile | 23m 26s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 26s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 7 new + 1813 unchanged - 51 fixed = 1820 total (was 1864) |
| +1 :green_heart: | compile | 20m 42s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 42s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 7 new + 1688 unchanged - 51 fixed = 1695 total (was 1739) |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/artifact/out/blanks-eol.txt) | The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 47s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/artifact/out/results-checkstyle-root.txt) | root: The patch generated 8 new + 71 unchanged - 84 fixed = 79 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 41s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 40s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 38s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 35s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 27s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 23m 54s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 27s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 17m 36s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 30s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 5s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not generate ASF License warnings. |
| | | 287m 27s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 9264e1b900a8 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 21647b2712100b0c8cb86ceed3f9513328d41268 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/testReport/ |
| Max. process+thread count | 3134 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r783920317
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -3821,23 +3481,14 @@ S3AFileStatus s3GetFileStatus(final Path path,
// by the time of listing so that the response includes some
// which have not.
- int listSize;
- if (tombstones == null) {
- // no tombstones so look for a marker and at least one child.
- listSize = 2;
- } else {
- // build a listing > tombstones. If the caller has many thousands
- // of tombstones this won't work properly, which is why pruning
- // of expired tombstones matters.
- listSize = Math.min(2 + tombstones.size(), Math.max(2, maxKeys));
- }
+ final int listSize = 2;
S3ListRequest request = createListObjectsRequest(dirKey, "/",
listSize);
// execute the request
S3ListResult listResult = listObjects(request,
getDurationTrackerFactory());
- if (listResult.hasPrefixesOrObjects(contextAccessors, tombstones)) {
+ if (listResult.hasPrefixesOrObjects(contextAccessors, null)) {
Review comment:
well spotted!
i've just gone through looking for all refs to *tombstone*
we can cut all the list reconciliation out, but also various test bits set up to generate unique paths and so avoid tombstone problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012520863
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 110 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 17s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 39s | | trunk passed |
| +1 :green_heart: | compile | 24m 30s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 7s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 57s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 43s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 20s | | the patch passed |
| +1 :green_heart: | compile | 23m 57s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 57s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/18/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51 fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 20m 52s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 52s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/18/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688 unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 58s | | root: The patch generated 0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 20s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 42s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 48s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 46s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 36s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 21m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 38s | | hadoop-project in the patch passed. |
| -1 :x: | unit | 18m 1s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/18/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 18s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 16s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not generate ASF License warnings. |
| | | 247m 1s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ipc.TestIPC |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux cdba0a50693f 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ac185db64d6e8c78178fb52b46f0f7b5c92344ab |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/18/testReport/ |
| Max. process+thread count | 1500 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/18/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014526054
rebased to trunk to line everything up for a merge and to retest locally first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bogthe commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
bogthe commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r732739468
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -3587,116 +3399,13 @@ S3AFileStatus innerGetFileStatus(final Path f,
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({}); needEmptyDirectory={}",
path, key, needEmptyDirectoryFlag);
+ // there was no entry in S3Guard
+ // retrieve the data and update the metadata store in the process.
Review comment:
Does this still update the metadata store? If not, the description of this method can be updated too since it mentions S3Guard too
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -494,27 +497,25 @@ private Constants() {
public static final String CUSTOM_SIGNERS = "fs.s3a.custom.signers";
/**
- * There's 3 parameters that can be used to specify a non-default signing
+ * Multiple can be used to specify a non-default signing
Review comment:
nit: + `parameters`
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -554,7 +439,7 @@ public void checkLocatedFileStatusScanFile() throws Throwable {
true,
TEXT_FILE,
true);
- accessDeniedIf(!guardedInAuthMode,
+ accessDeniedIf(!false,
Review comment:
nit: `!false` => `true`
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "",
+or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers, for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile, overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB: path names
-and summary details of objects —the actual data is stored in S3, so billed at S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests, combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in `core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0 --write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and `fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
Review comment:
Continuation missing?
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -494,27 +497,25 @@ private Constants() {
public static final String CUSTOM_SIGNERS = "fs.s3a.custom.signers";
/**
- * There's 3 parameters that can be used to specify a non-default signing
+ * Multiple can be used to specify a non-default signing
* algorithm.<br>
* fs.s3a.signing-algorithm - This property has existed for the longest time.
- * If specified, without either of the other 2 properties being specified,
- * this signing algorithm will be used for S3 and DDB (S3Guard). <br>
- * The other 2 properties override this value for S3 or DDB. <br>
+ * If specified, without other properties being specified,
+ * this signing algorithm will be used for all services. <br>
+ * Another property overridea this value for S3. <br>
Review comment:
nit: `overridea` -> `overrides`
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -456,21 +349,13 @@ public void checkGlobOperations() throws Throwable {
// baseline: the real filesystem on a subdir
globFS(getFileSystem(), subdirFile, null, false, 1);
// a file fails if not in auth mode
- globFS(readonlyFS, subdirFile, null, !guardedInAuthMode, 1);
+ globFS(readonlyFS, subdirFile, null, !false, 1);
Review comment:
nit: `!false` => `true`
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -422,7 +315,7 @@ public void checkBasicFileOperations() throws Throwable {
readonlyFS.getFileStatus(emptyDir);
// now look at a file; the outcome depends on the mode.
- accessDeniedIf(!guardedInAuthMode, () ->
+ accessDeniedIf(!false, () ->
Review comment:
nit: `!false` => `true`
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
##########
@@ -18,1003 +18,85 @@
package org.apache.hadoop.fs.s3a.s3guard;
-import java.io.FileNotFoundException;
-import java.io.IOException;
import java.net.URI;
import java.util.ArrayList;
import java.util.Collection;
-import java.util.Collections;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Set;
-import java.util.concurrent.TimeUnit;
import java.util.function.Function;
-import java.util.stream.Collectors;
-
-import javax.annotation.Nullable;
import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.thirdparty.com.google.common.base.Preconditions;
-import org.apache.hadoop.fs.RemoteIterator;
-import org.apache.hadoop.fs.s3a.S3AFileSystem;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.s3a.Retries;
-import org.apache.hadoop.fs.s3a.Retries.RetryTranslated;
-import org.apache.hadoop.fs.s3a.S3AFileStatus;
-import org.apache.hadoop.io.IOUtils;
-import org.apache.hadoop.util.ExitUtil;
-import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.s3a.Constants;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
-import static org.apache.hadoop.fs.s3a.Constants.*;
+import static org.apache.hadoop.fs.s3a.Constants.AUTHORITATIVE_PATH;
import static org.apache.hadoop.fs.s3a.Constants.DEFAULT_AUTHORITATIVE_PATH;
-import static org.apache.hadoop.fs.s3a.S3AUtils.createUploadFileStatus;
-import static org.apache.hadoop.fs.s3a.s3guard.PathMetadataDynamoDBTranslation.authoritativeEmptyDirectoryMarker;
-import static org.apache.hadoop.service.launcher.LauncherExitCodes.EXIT_BAD_CONFIGURATION;
/**
* Logic for integrating MetadataStore with S3A.
+ * Now that S3Guard is removed it is retained for the final
+ * end-of-S3Guard calls and when classes may be needed for
+ * compatibilty in external tests.
*/
+@SuppressWarnings("deprecation")
@InterfaceAudience.Private
@InterfaceStability.Unstable
public final class S3Guard {
private static final Logger LOG = LoggerFactory.getLogger(S3Guard.class);
- @InterfaceAudience.Private
- @InterfaceStability.Unstable
- @VisibleForTesting
- public static final String S3GUARD_DDB_CLIENT_FACTORY_IMPL =
- "fs.s3a.s3guard.ddb.client.factory.impl";
-
- static final Class<? extends DynamoDBClientFactory>
- S3GUARD_DDB_CLIENT_FACTORY_IMPL_DEFAULT =
- DynamoDBClientFactory.DefaultDynamoDBClientFactory.class;
- private static final S3AFileStatus[] EMPTY_LISTING = new S3AFileStatus[0];
-
- /**
- * Hard-coded policy : {@value}.
- * If true, when merging an S3 LIST with S3Guard in non-auth mode,
- * only updated entries are added; new entries are left out.
- * This policy choice reduces the amount of data stored in Dynamo,
- * and hence the complexity of the merge in a non-auth listing.
- */
- @VisibleForTesting
- public static final boolean DIR_MERGE_UPDATES_ALL_RECORDS_NONAUTH = false;
-
// Utility class. All static functions.
- private S3Guard() { }
-
- /* Utility functions. */
-
- /**
- * Create a new instance of the configured MetadataStore.
- * The returned MetadataStore will have been initialized via
- * {@link MetadataStore#initialize(FileSystem, ITtlTimeProvider)}
- * by this function before returning it. Callers must clean up by calling
- * {@link MetadataStore#close()} when done using the MetadataStore.
- *
- * @param fs FileSystem whose Configuration specifies which
- * implementation to use.
- * @param ttlTimeProvider
- * @return Reference to new MetadataStore.
- * @throws IOException if the metadata store cannot be instantiated
- */
- @Retries.OnceTranslated
- public static MetadataStore getMetadataStore(FileSystem fs,
- ITtlTimeProvider ttlTimeProvider)
- throws IOException {
- Preconditions.checkNotNull(fs);
- Configuration conf = fs.getConf();
- Preconditions.checkNotNull(conf);
- MetadataStore msInstance;
- try {
- Class<? extends MetadataStore> msClass = getMetadataStoreClass(conf);
- msInstance = ReflectionUtils.newInstance(msClass, conf);
- LOG.debug("Using {} metadata store for {} filesystem",
- msClass.getSimpleName(), fs.getScheme());
- msInstance.initialize(fs, ttlTimeProvider);
- return msInstance;
- } catch (FileNotFoundException e) {
- // Don't log this exception as it means the table doesn't exist yet;
- // rely on callers to catch and treat specially
- throw e;
- } catch (RuntimeException | IOException e) {
- String message = "Failed to instantiate metadata store " +
- conf.get(S3_METADATA_STORE_IMPL)
- + " defined in " + S3_METADATA_STORE_IMPL
- + ": " + e;
- LOG.error(message, e);
- if (e instanceof IOException) {
- throw e;
- } else {
- throw new IOException(message, e);
- }
- }
- }
-
- static Class<? extends MetadataStore> getMetadataStoreClass(
- Configuration conf) {
- if (conf == null) {
- return NullMetadataStore.class;
- }
- if (conf.get(S3_METADATA_STORE_IMPL) != null && LOG.isDebugEnabled()) {
- LOG.debug("Metastore option source {}",
- (Object)conf.getPropertySources(S3_METADATA_STORE_IMPL));
- }
-
- Class<? extends MetadataStore> aClass = conf.getClass(
- S3_METADATA_STORE_IMPL, NullMetadataStore.class,
- MetadataStore.class);
- return aClass;
- }
-
-
- /**
- * We update the metastore for the specific case of S3 value == S3Guard value
- * so as to place a more recent modtime in the store.
- * because if not, we will continue to probe S3 whenever we look for this
- * object, even we only do this if confident the S3 status is the same
- * as the one in the store (i.e. it is not an older version)
- * @param metadataStore MetadataStore to {@code put()} into.
- * @param pm current data
- * @param s3AFileStatus status to store
- * @param timeProvider Time provider to use when writing entries
- * @return true if the entry was updated.
- * @throws IOException if metadata store update failed
- */
- @RetryTranslated
- public static boolean refreshEntry(
- MetadataStore metadataStore,
- PathMetadata pm,
- S3AFileStatus s3AFileStatus,
- ITtlTimeProvider timeProvider) throws IOException {
- // the modtime of the data is the same as/older than the s3guard value
- // either an old object has been found, or the existing one was retrieved
- // in both cases -return s3guard value
- S3AFileStatus msStatus = pm.getFileStatus();
-
- // first check: size
- boolean sizeMatch = msStatus.getLen() == s3AFileStatus.getLen();
-
- // etags are expected on all objects, but handle the situation
- // that a third party store doesn't serve them.
- String s3Etag = s3AFileStatus.getETag();
- String pmEtag = msStatus.getETag();
- boolean etagsMatch = s3Etag != null && s3Etag.equals(pmEtag);
-
- // version ID: only in some stores, and will be missing in the metastore
- // if the entry was created through a list operation.
- String s3VersionId = s3AFileStatus.getVersionId();
- String pmVersionId = msStatus.getVersionId();
- boolean versionsMatchOrMissingInMetastore =
- pmVersionId == null || pmVersionId.equals(s3VersionId);
- if (sizeMatch && etagsMatch && versionsMatchOrMissingInMetastore) {
- // update the store, return the new value
- LOG.debug("Refreshing the metastore entry/timestamp");
- putAndReturn(metadataStore, s3AFileStatus, timeProvider);
- return true;
- }
- return false;
- }
-
- /**
- * Helper function which puts a given S3AFileStatus into the MetadataStore and
- * returns the same S3AFileStatus. Instrumentation monitors the put operation.
- * @param ms MetadataStore to {@code put()} into.
- * @param status status to store
- * @param timeProvider Time provider to use when writing entries
- * @return The same status as passed in
- * @throws IOException if metadata store update failed
- */
- @RetryTranslated
- public static S3AFileStatus putAndReturn(MetadataStore ms,
- S3AFileStatus status,
- ITtlTimeProvider timeProvider) throws IOException {
- return putAndReturn(ms, status, timeProvider, null);
- }
-
- /**
- * Helper function which puts a given S3AFileStatus into the MetadataStore and
- * returns the same S3AFileStatus. Instrumentation monitors the put operation.
- * @param ms MetadataStore to {@code put()} into.
- * @param status status to store
- * @param timeProvider Time provider to use when writing entries
- * @param operationState possibly-null metastore state tracker.
- * @return The same status as passed in
- * @throws IOException if metadata store update failed
- */
- @RetryTranslated
- public static S3AFileStatus putAndReturn(
- final MetadataStore ms,
- final S3AFileStatus status,
- final ITtlTimeProvider timeProvider,
- @Nullable final BulkOperationState operationState) throws IOException {
- long startTimeNano = System.nanoTime();
- try {
- putWithTtl(ms, new PathMetadata(status), timeProvider, operationState);
- } finally {
- ms.getInstrumentation().entryAdded((System.nanoTime() - startTimeNano));
- }
- return status;
- }
-
- /**
- * Creates an authoritative directory marker for the store.
- * @param ms MetadataStore to {@code put()} into.
- * @param status status to store
- * @param timeProvider Time provider to use when writing entries
- * @param operationState possibly-null metastore state tracker.
- * @throws IOException if metadata store update failed
- */
- @RetryTranslated
- public static void putAuthDirectoryMarker(
- final MetadataStore ms,
- final S3AFileStatus status,
- final ITtlTimeProvider timeProvider,
- @Nullable final BulkOperationState operationState) throws IOException {
- long startTimeNano = System.nanoTime();
- try {
- final PathMetadata fileMeta = authoritativeEmptyDirectoryMarker(status);
- putWithTtl(ms, fileMeta, timeProvider, operationState);
- } finally {
- ms.getInstrumentation().directoryMarkedAuthoritative();
- ms.getInstrumentation().entryAdded((System.nanoTime() - startTimeNano));
- }
- }
-
- /**
- * Initiate a bulk write and create an operation state for it.
- * This may then be passed into put operations.
- * @param metastore store
- * @param operation the type of the operation.
- * @param path path under which updates will be explicitly put.
- * @return a store-specific state to pass into the put operations, or null
- * @throws IOException failure
- */
- public static BulkOperationState initiateBulkWrite(
- @Nullable final MetadataStore metastore,
- final BulkOperationState.OperationType operation,
- final Path path) throws IOException {
- Preconditions.checkArgument(
- operation != BulkOperationState.OperationType.Rename,
- "Rename operations cannot be started through initiateBulkWrite");
- if (metastore == null || isNullMetadataStore(metastore)) {
- return null;
- } else {
- return metastore.initiateBulkWrite(operation, path);
- }
- }
-
- /**
- * Convert the data of a directory listing to an array of {@link FileStatus}
- * entries. Tombstones are filtered out at this point. If the listing is null
- * an empty array is returned.
- * @param dirMeta directory listing -may be null
- * @return a possibly-empty array of file status entries
- */
- public static S3AFileStatus[] dirMetaToStatuses(DirListingMetadata dirMeta) {
- if (dirMeta == null) {
- return EMPTY_LISTING;
- }
-
- Collection<PathMetadata> listing = dirMeta.getListing();
- List<FileStatus> statuses = new ArrayList<>();
-
- for (PathMetadata pm : listing) {
- if (!pm.isDeleted()) {
- statuses.add(pm.getFileStatus());
- }
- }
-
- return statuses.toArray(new S3AFileStatus[0]);
- }
-
- /**
- * Given directory listing metadata from both the backing store and the
- * MetadataStore, merge the two sources of truth to create a consistent
- * view of the current directory contents, which can be returned to clients.
- *
- * Also update the MetadataStore to reflect the resulting directory listing.
- *
- * In not authoritative case: update file metadata if mod_time in listing
- * of a file is greater then what is currently in the ms
- *
- * @param ms MetadataStore to use.
- * @param path path to directory
- * @param backingStatuses Directory listing from the backing store.
- * @param dirMeta Directory listing from MetadataStore. May be null.
- * @param isAuthoritative State of authoritative mode
- * @param timeProvider Time provider to use when updating entries
- * @param toStatusItr function to convert array of file status to
- * RemoteIterator.
- * @return Final result of directory listing.
- * @throws IOException if metadata store update failed
- */
- public static RemoteIterator<S3AFileStatus> dirListingUnion(
- MetadataStore ms, Path path,
- RemoteIterator<S3AFileStatus> backingStatuses,
- DirListingMetadata dirMeta, boolean isAuthoritative,
- ITtlTimeProvider timeProvider,
- Function<S3AFileStatus[], RemoteIterator<S3AFileStatus>> toStatusItr)
- throws IOException {
-
- // Fast-path for NullMetadataStore
- if (isNullMetadataStore(ms)) {
- return backingStatuses;
- }
-
- assertQualified(path);
-
- if (dirMeta == null) {
- // The metadataStore had zero state for this directory
- dirMeta = new DirListingMetadata(path, DirListingMetadata.EMPTY_DIR,
- false);
- }
-
- // Since we treat the MetadataStore as a "fresher" or "consistent" view
- // of metadata, we always use its metadata first.
-
- // Since the authoritative case is already handled outside this function,
- // we will basically start with the set of directory entries in the
- // DirListingMetadata, and add any that only exist in the backingStatuses.
- //
- // We try to avoid writing any more child entries than need be to :-
- // (a) save time and money.
- // (b) avoid overwriting the authoritative bit of children (HADOOP-16746).
- // For auth mode updates, we supply the full listing and a list of which
- // child entries have not been changed; the store gets to optimize its
- // update however it chooses.
- //
- // for non-auth-mode S3Guard, we just build a list of entries to add and
- // submit them in a batch; this is more efficient than trickling out the
- // updates one-by-one.
-
- BulkOperationState operationState = ms.initiateBulkWrite(
- BulkOperationState.OperationType.Listing,
- path);
- if (isAuthoritative) {
- authoritativeUnion(ms, path, backingStatuses, dirMeta,
- timeProvider, operationState);
- } else {
- nonAuthoritativeUnion(ms, path, backingStatuses, dirMeta,
- timeProvider, operationState);
- }
- IOUtils.cleanupWithLogger(LOG, operationState);
-
- return toStatusItr.apply(dirMetaToStatuses(dirMeta));
+ private S3Guard() {
}
/**
- * Perform the authoritative union operation.
- * Here all updated/missing entries are added back; we take care
- * not to overwrite unchanged entries as that will lose their
- * isAuthoritative bit (HADOOP-16746).
- * @param ms MetadataStore to use.
- * @param path path to directory
- * @param backingStatuses Directory listing from the backing store.
- * @param dirMeta Directory listing from MetadataStore. May be null.
- * @param timeProvider Time provider to use when updating entries
- * @param operationState ongoing operation
- * @throws IOException if metadata store update failed
- */
- private static void authoritativeUnion(
- final MetadataStore ms,
- final Path path,
- final RemoteIterator<S3AFileStatus> backingStatuses,
- final DirListingMetadata dirMeta,
- final ITtlTimeProvider timeProvider,
- final BulkOperationState operationState) throws IOException {
- // track all unchanged entries; used so the metastore can identify entries
- // it doesn't need to update
- List<Path> unchangedEntries = new ArrayList<>(dirMeta.getListing().size());
- boolean changed = !dirMeta.isAuthoritative();
- Set<Path> deleted = dirMeta.listTombstones();
- final Map<Path, PathMetadata> dirMetaMap = dirMeta.getListing().stream()
- .collect(Collectors.toMap(pm -> pm.getFileStatus().getPath(), pm -> pm));
- while (backingStatuses.hasNext()) {
- S3AFileStatus s = backingStatuses.next();
- final Path statusPath = s.getPath();
- if (deleted.contains(statusPath)) {
- continue;
- }
-
- // this is built up to be whatever entry is to be added to the dirMeta
- // collection
- PathMetadata pathMetadata = dirMetaMap.get(statusPath);
-
- if (pathMetadata == null) {
- // there's no entry in the listing, so create one.
- pathMetadata = new PathMetadata(s);
- } else {
- // no change -add the path to the list of unchangedEntries
- unchangedEntries.add(statusPath);
- }
-
- // Minor race condition here. Multiple threads could add to this
- // mutable DirListingMetadata. Since it is backed by a
- // ConcurrentHashMap, the last put() wins.
- // More concerning is two threads racing on listStatus() and delete().
- // Any FileSystem has similar race conditions, but we could persist
- // a stale entry longer. We could expose an atomic
- // DirListingMetadata#putIfNotPresent()
- changed |= dirMeta.put(pathMetadata);
- }
-
- if (changed) {
- // in an authoritative update, we pass in the full list of entries,
- // but do declare which have not changed to avoid needless and potentially
- // destructive overwrites.
- LOG.debug("Marking the directory {} as authoritative", path);
- ms.getInstrumentation().directoryMarkedAuthoritative();
- dirMeta.setAuthoritative(true); // This is the full directory contents
- // write the updated dir entry and any changed children.
- S3Guard.putWithTtl(ms, dirMeta, unchangedEntries, timeProvider, operationState);
- }
- }
-
- /**
- * Perform the authoritative union operation.
- * @param ms MetadataStore to use.
- * @param path path to directory
- * @param backingStatuses Directory listing from the backing store.
- * @param dirMeta Directory listing from MetadataStore. May be null.
- * @param timeProvider Time provider to use when updating entries
- * @param operationState ongoing operation
- * @throws IOException if metadata store update failed
- */
- private static void nonAuthoritativeUnion(
- final MetadataStore ms,
- final Path path,
- final RemoteIterator<S3AFileStatus> backingStatuses,
- final DirListingMetadata dirMeta,
- final ITtlTimeProvider timeProvider,
- final BulkOperationState operationState) throws IOException {
- List<PathMetadata> entriesToAdd = new ArrayList<>();
- Set<Path> deleted = dirMeta.listTombstones();
-
- final Map<Path, PathMetadata> dirMetaMap = dirMeta.getListing().stream()
- .collect(Collectors.toMap(pm -> pm.getFileStatus().getPath(), pm -> pm));
- while (backingStatuses.hasNext()) {
- S3AFileStatus s = backingStatuses.next();
- final Path statusPath = s.getPath();
- if (deleted.contains(statusPath)) {
- continue;
- }
-
- // this is the record in dynamo
- PathMetadata pathMetadata = dirMetaMap.get(statusPath);
-
- // in non-auth listings, we compare the file status of the metastore
- // list with those in the FS, and overwrite the MS entry if
- // either of two conditions are met
- // - there is no entry in the metastore and
- // DIR_MERGE_UPDATES_ALL_RECORDS_NONAUTH is compiled to true
- // - there is an entry in the metastore the FS entry is newer.
- boolean shouldUpdate;
- if (pathMetadata != null) {
- // entry is in DDB; check modification time
- shouldUpdate = s.getModificationTime() > (pathMetadata.getFileStatus())
- .getModificationTime();
- // create an updated record.
- pathMetadata = new PathMetadata(s);
- } else {
- // entry is not present. Create for insertion into dirMeta
- pathMetadata = new PathMetadata(s);
- // use hard-coded policy about updating
- shouldUpdate = DIR_MERGE_UPDATES_ALL_RECORDS_NONAUTH;
- }
- if (shouldUpdate) {
- // we do want to update DDB and the listing with a new entry.
- LOG.debug("Update ms with newer metadata of: {}", s);
- // ensure it gets into the dirListing
- // add to the list of entries to add later,
- entriesToAdd.add(pathMetadata);
- }
- // add the entry to the union; no-op if it was already there.
- dirMeta.put(pathMetadata);
- }
-
- if (!entriesToAdd.isEmpty()) {
- // non-auth, just push out the updated entry list
- LOG.debug("Adding {} entries under directory {}", entriesToAdd.size(), path);
- putWithTtl(ms, entriesToAdd, timeProvider, operationState);
- }
- }
-
- /**
- * Although NullMetadataStore does nothing, callers may wish to avoid work
- * (fast path) when the NullMetadataStore is in use.
- * @param ms The MetadataStore to test
- * @return true iff the MetadataStore is the null, or no-op, implementation.
- */
- public static boolean isNullMetadataStore(MetadataStore ms) {
- return (ms instanceof NullMetadataStore);
- }
-
- /**
- * Update MetadataStore to reflect creation of the given directories.
- *
- * If an IOException is raised while trying to update the entry, this
- * operation catches the exception, swallows it and returns.
- *
- * @deprecated this is no longer called by {@code S3AFilesystem.innerMkDirs}.
- * See: HADOOP-15079 (January 2018).
- * It is currently retained because of its discussion in the method on
- * atomicity and in case we need to reinstate it or adapt the current
- * process of directory marker creation.
- * But it is not being tested and so may age with time...consider
- * deleting it in future if it's clear there's no need for it.
- * @param ms MetadataStore to update.
- * @param dirs null, or an ordered list of directories from leaf to root.
- * E.g. if /a/ exists, and mkdirs(/a/b/c/d) is called, this
- * list will contain [/a/b/c/d, /a/b/c, /a/b]. /a/b/c/d is
- * an empty, dir, and the other dirs only contain their child
- * dir.
- * @param owner Hadoop user name.
- * @param authoritative Whether to mark new directories as authoritative.
- * @param timeProvider Time provider.
- */
- @Deprecated
- @Retries.OnceExceptionsSwallowed
- public static void makeDirsOrdered(MetadataStore ms, List<Path> dirs,
- String owner, boolean authoritative, ITtlTimeProvider timeProvider) {
- if (dirs == null) {
- return;
- }
-
- /* We discussed atomicity of this implementation.
- * The concern is that multiple clients could race to write different
- * cached directories to the MetadataStore. Two solutions are proposed:
- * 1. Move mkdirs() into MetadataStore interface and let implementations
- * ensure they are atomic.
- * 2. Specify that the semantics of MetadataStore#putListStatus() is
- * always additive, That is, if MetadataStore has listStatus() state
- * for /a/b that contains [/a/b/file0, /a/b/file1], and we then call
- * putListStatus(/a/b -> [/a/b/file2, /a/b/file3], isAuthoritative=true),
- * then we will end up with final state of
- * [/a/b/file0, /a/b/file1, /a/b/file2, /a/b/file3], isAuthoritative =
- * true
- */
- S3AFileStatus prevStatus = null;
-
- // Use new batched put to reduce round trips.
- List<PathMetadata> pathMetas = new ArrayList<>(dirs.size());
-
- try {
- // Iterate from leaf to root
- for (int i = 0; i < dirs.size(); i++) {
- boolean isLeaf = (prevStatus == null);
- Path f = dirs.get(i);
- assertQualified(f);
- S3AFileStatus status =
- createUploadFileStatus(f, true, 0, 0, owner, null, null);
-
- // We only need to put a DirListingMetadata if we are setting
- // authoritative bit
- DirListingMetadata dirMeta = null;
- if (authoritative) {
- Collection<PathMetadata> children;
- if (isLeaf) {
- children = DirListingMetadata.EMPTY_DIR;
- } else {
- children = new ArrayList<>(1);
- children.add(new PathMetadata(prevStatus));
- }
- dirMeta = new DirListingMetadata(f, children, authoritative);
- S3Guard.putWithTtl(ms, dirMeta, Collections.emptyList(), timeProvider, null);
- }
-
- pathMetas.add(new PathMetadata(status));
- prevStatus = status;
- }
-
- // Batched put
- S3Guard.putWithTtl(ms, pathMetas, timeProvider, null);
- } catch (IOException ioe) {
- LOG.error("MetadataStore#put() failure:", ioe);
- }
- }
-
- /**
- * Helper function that records the move of directory paths, adding
- * resulting metadata to the supplied lists.
- * Does not store in MetadataStore.
- * @param ms MetadataStore, used to make this a no-op, when it is
- * NullMetadataStore.
- * @param srcPaths stores the source path here
- * @param dstMetas stores destination metadata here
- * @param srcPath source path to store
- * @param dstPath destination path to store
- * @param owner file owner to use in created records
- */
- public static void addMoveDir(MetadataStore ms, Collection<Path> srcPaths,
- Collection<PathMetadata> dstMetas, Path srcPath, Path dstPath,
- String owner) {
- if (isNullMetadataStore(ms)) {
- return;
- }
- assertQualified(srcPath, dstPath);
-
- S3AFileStatus dstStatus = createUploadFileStatus(dstPath, true, 0,
- 0, owner, null, null);
- addMoveStatus(srcPaths, dstMetas, srcPath, dstStatus);
- }
-
- /**
- * Like {@link #addMoveDir(MetadataStore, Collection, Collection, Path,
- * Path, String)} (), but for files.
- * @param ms MetadataStore, used to make this a no-op, when it is
- * NullMetadataStore.
- * @param srcPaths stores the source path here
- * @param dstMetas stores destination metadata here
- * @param srcPath source path to store
- * @param dstPath destination path to store
- * @param size length of file moved
- * @param blockSize blocksize to associate with destination file
- * @param owner file owner to use in created records
- * @param eTag the s3 object eTag of file moved
- * @param versionId the s3 object versionId of file moved
- */
- public static void addMoveFile(MetadataStore ms, Collection<Path> srcPaths,
- Collection<PathMetadata> dstMetas, Path srcPath, Path dstPath,
- long size, long blockSize, String owner, String eTag, String versionId) {
- if (isNullMetadataStore(ms)) {
- return;
- }
- assertQualified(srcPath, dstPath);
- S3AFileStatus dstStatus = createUploadFileStatus(dstPath, false,
- size, blockSize, owner, eTag, versionId);
- addMoveStatus(srcPaths, dstMetas, srcPath, dstStatus);
- }
-
- /**
- * Helper method that records the move of all ancestors of a path.
- *
- * In S3A, an optimization is to delete unnecessary fake directory objects if
- * the directory is non-empty. In that case, for a nested child to move, S3A
- * is not listing and thus moving all its ancestors (up to source root). So we
- * take care of those inferred directories of this path explicitly.
- *
- * As {@link #addMoveFile} and {@link #addMoveDir}, this method adds resulting
- * metadata to the supplied lists. It does not update the MetadataStore.
- *
- * @param ms MetadataStore, no-op if it is NullMetadataStore
- * @param srcPaths stores the source path here
- * @param dstMetas stores destination metadata here
- * @param srcRoot source root up to which (exclusive) should we add ancestors
- * @param srcPath source path of the child to add ancestors
- * @param dstPath destination path of the child to add ancestors
- * @param owner Hadoop user name
- */
- public static void addMoveAncestors(MetadataStore ms,
- Collection<Path> srcPaths, Collection<PathMetadata> dstMetas,
- Path srcRoot, Path srcPath, Path dstPath, String owner) {
- if (isNullMetadataStore(ms)) {
- return;
- }
-
- assertQualified(srcRoot, srcPath, dstPath);
-
- if (srcPath.equals(srcRoot)) {
- LOG.debug("Skip moving ancestors of source root directory {}", srcRoot);
- return;
- }
-
- Path parentSrc = srcPath.getParent();
- Path parentDst = dstPath.getParent();
- while (parentSrc != null
- && !parentSrc.isRoot()
- && !parentSrc.equals(srcRoot)
- && !srcPaths.contains(parentSrc)) {
- LOG.debug("Renaming non-listed parent {} to {}", parentSrc, parentDst);
- S3Guard.addMoveDir(ms, srcPaths, dstMetas, parentSrc, parentDst, owner);
- parentSrc = parentSrc.getParent();
- parentDst = parentDst.getParent();
- }
- }
-
- /**
- * This adds all new ancestors of a path as directories.
- * This forwards to
- * {@link MetadataStore#addAncestors(Path, BulkOperationState)}.
- * <p>
- * Originally it implemented the logic to probe for an add ancestors,
- * but with the addition of a store-specific bulk operation state
- * it became unworkable.
- *
- * @param metadataStore store
- * @param qualifiedPath path to update
- * @param operationState (nullable) operational state for a bulk update
- * @throws IOException failure
- */
- @Retries.RetryTranslated
- public static void addAncestors(
- final MetadataStore metadataStore,
- final Path qualifiedPath,
- final ITtlTimeProvider timeProvider,
- @Nullable final BulkOperationState operationState) throws IOException {
- metadataStore.addAncestors(qualifiedPath, operationState);
- }
-
- /**
- * Add the fact that a file was moved from a source path to a destination.
- * @param srcPaths collection of source paths to update
- * @param dstMetas collection of destination meta data entries to update.
- * @param srcPath path of the source file.
- * @param dstStatus status of the source file after it was copied.
- */
- private static void addMoveStatus(Collection<Path> srcPaths,
- Collection<PathMetadata> dstMetas,
- Path srcPath,
- S3AFileStatus dstStatus) {
- srcPaths.add(srcPath);
- dstMetas.add(new PathMetadata(dstStatus));
- }
-
- /**
- * Assert that the path is qualified with a host and scheme.
- * @param p path to check
- * @throws NullPointerException if either argument does not hold
- */
- public static void assertQualified(Path p) {
- URI uri = p.toUri();
- // Paths must include bucket in case MetadataStore is shared between
- // multiple S3AFileSystem instances
- Preconditions.checkNotNull(uri.getHost(), "Null host in " + uri);
-
- // This should never fail, but is retained for completeness.
- Preconditions.checkNotNull(uri.getScheme(), "Null scheme in " + uri);
- }
-
- /**
- * Assert that all paths are valid.
- * @param paths path to check
- * @throws NullPointerException if either argument does not hold
+ * Assert that there is no S3Guard.
+ * @param fsURI FileSystem URI
+ * @param conf configuration
*/
- public static void assertQualified(Path...paths) {
- for (Path path : paths) {
- assertQualified(path);
+ public static void checkNoS3Guard(URI fsURI, Configuration conf) throws PathIOException {
+ final String classname = conf.getTrimmed(Constants.S3_METADATA_STORE_IMPL, "");
+ if (!"org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore".equals(classname)
+ && !classname.isEmpty()) {
+ LOG.error("Metastore option source {}",
+ (Object) conf.getPropertySources(Constants.S3_METADATA_STORE_IMPL));
+ throw new PathIOException(fsURI.toString(),
+ "Filesystem is configured to use Unsupported S3Guard store " + classname);
}
}
/**
* Runtime implementation for TTL Time Provider interface.
+ * This is deprecated; retained pnly because the StoreContext builder takes it
Review comment:
nit: `pnly` -> `only`
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -437,7 +330,7 @@ public void checkBasicFileOperations() throws Throwable {
// This means that permissions on the file do not get checked.
// See: HADOOP-16464.
Optional<FSDataInputStream> optIn = accessDeniedIf(
- !guardedInAuthMode, () -> readonlyFS.open(emptyFile));
+ !false, () -> readonlyFS.open(emptyFile));
Review comment:
nit: `!false` => `true`
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "",
+or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers, for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile, overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB: path names
-and summary details of objects —the actual data is stored in S3, so billed at S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests, combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in `core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0 --write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and `fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
-```
-hadoop s3guard import
-
-import [OPTIONS] [s3a://PATH]
- import metadata from existing S3 data
-
-Common options:
- -authoritative - Mark imported directory data as authoritative.
- -verbose - Verbose Output.
- -meta URL - Metadata repository details (implementation-specific)
-
-Amazon DynamoDB-specific options:
- -region REGION - Service region for connections
-
- URLs for Amazon DynamoDB are of the form dynamodb://TABLE_NAME.
- Specifying both the -region option and an S3A path
- is not supported.
-```
-
-Example
-
-Import all files and directories in a bucket into the S3Guard table.
-
-```bash
-hadoop s3guard import s3a://ireland-1
-```
-
-Import a directory tree, marking directories as authoritative.
-
-```bash
-hadoop s3guard import -authoritative -verbose s3a://ireland-1/fork-0008
-
-2020-01-03 12:05:18,321 [main] INFO - Metadata store DynamoDBMetadataStore{region=eu-west-1,
- tableName=s3guard-metadata, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/s3guard-metadata} is initialized.
-2020-01-03 12:05:18,324 [main] INFO - Starting: Importing s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,324 [main] INFO - Importing directory s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,537 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-false
-2020-01-03 12:05:18,630 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-true
-2020-01-03 12:05:19,142 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-0
-2020-01-03 12:05:19,191 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-1
-2020-01-03 12:05:19,240 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-0
-2020-01-03 12:05:19,289 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-1
-2020-01-03 12:05:19,314 [main] INFO - Updated S3Guard with 0 files and 6 directory entries
-2020-01-03 12:05:19,315 [main] INFO - Marking directory tree s3a://ireland-1/fork-0008 as authoritative
-2020-01-03 12:05:19,342 [main] INFO - Importing s3a://ireland-1/fork-0008: duration 0:01.018s
-Inserted 6 items into Metadata Store
-```
-
-### Compare a S3Guard table and the S3 Store: `s3guard diff`
-
-```bash
-hadoop s3guard diff [-meta URI] s3a://BUCKET
-```
-
-Lists discrepancies between a metadata store and bucket. Note that depending on
-how S3Guard is used, certain discrepancies are to be expected.
-
-Example
-
-```bash
-hadoop s3guard diff s3a://ireland-1
```
Review comment:
Needs to be deleted too probably
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentS3Object.java
##########
@@ -173,7 +173,7 @@ private void skipFailpoint(long len) throws IOException {
}
private boolean shouldInjectFailure(String key) {
- if (policy.shouldDelay(key) &&
+ if (false &&
readFailureCounter < MAX_READ_FAILURES) {
readFailureCounter++;
return true;
Review comment:
Always returns false so probably remove?
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/delegation_tokens.md
##########
@@ -39,8 +39,7 @@ Clients with this token have the full permissions of the user.
*Role Delegation Tokens:* These contain an "STS Session Token" requested by by the
STS "Assume Role" API, so grant the caller to interact with S3 as specific AWS
-role, *with permissions restricted to purely accessing the S3 bucket
-and associated S3Guard data*.
+role, *with permissions restricted to purely accessing the S3 bucket.
Review comment:
nit: missing `*` at the end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-962095446
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 106 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 22s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 22s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 19s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 0m 19s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 18s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 20 new + 71 unchanged - 83 fixed = 91 total (was 154) |
| -1 :x: | mvnsite | 0m 21s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 24s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 19s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 23m 18s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 23s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 90m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle markdownlint |
| uname | Linux 91e6eb1145bb 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2dd5c1995c7e5cc20f4acf7e29a45ee131d8cb02 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/testReport/ |
| Max. process+thread count | 625 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012520838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011191687
`[ERROR] Failures:
[ERROR] ITestS3AMiscOperations.testEmptyFileChecksums:182->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums of empty files expected:<etag: "5334c7a47521faa37aaaadb3eda3cc7c"> but was:<etag: "37d542d4dade6ea73842e723b57b4c9b">
[ERROR] ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted:258->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums expected:<etag: "057988c6b2a0f57adf8c83c695059978"> but was:<etag: "7ecdd0d908e71fc933749af8612460d7">
[ERROR] Errors: `
Also these failures not sure if they are related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-938643502
@mehakmeet @mukund-thakur @bogthe @bgaborg
going to be a big patch, but it removes everything.
Current PR turns s3guard off, but doesn't yet remove the (no always no-op API calls) from the listing/delete/rename code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r782546599
##########
File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
##########
@@ -343,7 +343,7 @@ stores pretend that they are a FileSystem, a FileSystem with the same
features and operations as HDFS. This is —ultimately—a pretence:
they have different characteristics and occasionally the illusion fails.
-1. **Consistency**. Object stores are generally *Eventually Consistent*: it
+1. **Consistency**. Object mqy be *Eventually Consistent*: it
Review comment:
fixed. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011225626
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 108 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 42s | | trunk passed |
| +1 :green_heart: | compile | 24m 16s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 34s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 49s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 19s | | the patch passed |
| +1 :green_heart: | compile | 28m 14s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 28m 14s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 4 new + 1813 unchanged - 51 fixed = 1817 total (was 1864) |
| +1 :green_heart: | compile | 22m 36s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 22m 36s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 4 new + 1688 unchanged - 51 fixed = 1692 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 54s | | root: The patch generated 0 new + 71 unchanged - 84 fixed = 71 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 45s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 38s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 39s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 31s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 23m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | | hadoop-project in the patch passed. |
| -1 :x: | unit | 18m 9s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 18s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 20s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. |
| | | 251m 51s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ipc.TestIPC |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux dece1db4343d 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 005dcfab3004db8c679456731ff42911a0ef4550 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011359446
new pr up; testing s3 london. `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=delete -Dscale`. functional but slow (!)
```
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 44:39 min (Wall Clock)
[INFO] Finished at: 2022-01-12T16:29:50Z
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r753789722
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java
##########
@@ -60,12 +55,13 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
/**
* A wrapper around {@link com.amazonaws.services.s3.AmazonS3} that injects
- * inconsistency and/or errors. Used for testing S3Guard.
- * Currently only delays listing visibility, not affecting GET.
+ * failures.
+ * It used to also inject inconsistency, but this was removed with S3Guard;
+ * what is retained is the ability to throttle AWS operations and for the
+ * input stream to be inconsistent.
*/
@InterfaceAudience.Private
@InterfaceStability.Unstable
Review comment:
keeping failure injection as it can set read failures and throttling, but cutting paths for delayed listing, the settings/probes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-969227284
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 10s | | trunk passed |
| +1 :green_heart: | compile | 28m 7s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 30s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 23s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 25m 45s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 25m 45s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 22m 18s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 22m 18s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 11s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/results-checkstyle-root.txt) | root: The patch generated 15 new + 71 unchanged - 83 fixed = 86 total (was 154) |
| -1 :x: | mvnsite | 0m 40s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 4s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 43s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 38s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 27m 17s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 51s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 39s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not generate ASF License warnings. |
| | | 247m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle |
| uname | Linux c79218c1d8fe 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2eced9dd71c1ac4bacee97f92e0c63a29db3faae |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r765877448
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java
##########
@@ -279,82 +205,19 @@ private ObjectListingIterator createObjectListingIterator(
LOG.debug("Requesting all entries under {} with delimiter '{}'",
key, delimiter);
}
- final RemoteIterator<S3AFileStatus> cachedFilesIterator;
- final Set<Path> tombstones;
- boolean allowAuthoritative = listingOperationCallbacks
- .allowAuthoritative(path);
- if (recursive) {
- final PathMetadata pm = getStoreContext()
- .getMetadataStore()
- .get(path, true);
- if (pm != null) {
- if (pm.isDeleted()) {
- OffsetDateTime deletedAt = OffsetDateTime
- .ofInstant(Instant.ofEpochMilli(
- pm.getFileStatus().getModificationTime()),
- ZoneOffset.UTC);
- throw new FileNotFoundException("Path " + path + " is recorded as " +
- "deleted by S3Guard at " + deletedAt);
- }
- }
- MetadataStoreListFilesIterator metadataStoreListFilesIterator =
- new MetadataStoreListFilesIterator(
- getStoreContext().getMetadataStore(),
- pm,
- allowAuthoritative);
- tombstones = metadataStoreListFilesIterator.listTombstones();
- // if all of the below is true
- // - authoritative access is allowed for this metadatastore
- // for this directory,
- // - all the directory listings are authoritative on the client
- // - the caller does not force non-authoritative access
- // return the listing without any further s3 access
- if (!forceNonAuthoritativeMS &&
- allowAuthoritative &&
- metadataStoreListFilesIterator.isRecursivelyAuthoritative()) {
- S3AFileStatus[] statuses = S3AUtils.iteratorToStatuses(
- metadataStoreListFilesIterator, tombstones);
- cachedFilesIterator = createProvidedFileStatusIterator(
- statuses, ACCEPT_ALL, acceptor);
- return createLocatedFileStatusIterator(cachedFilesIterator);
- }
- cachedFilesIterator = metadataStoreListFilesIterator;
- } else {
- DirListingMetadata meta =
- S3Guard.listChildrenWithTtl(
- getStoreContext().getMetadataStore(),
- path,
- listingOperationCallbacks.getUpdatedTtlTimeProvider(),
- allowAuthoritative);
- if (meta != null) {
- tombstones = meta.listTombstones();
- } else {
- tombstones = null;
- }
- cachedFilesIterator = createProvidedFileStatusIterator(
- S3Guard.dirMetaToStatuses(meta), ACCEPT_ALL, acceptor);
- if (allowAuthoritative && meta != null && meta.isAuthoritative()) {
- // metadata listing is authoritative, so return it directly
- return createLocatedFileStatusIterator(cachedFilesIterator);
- }
- }
- return createTombstoneReconcilingIterator(
- createLocatedFileStatusIterator(
- createFileStatusListingIterator(path,
- listingOperationCallbacks
- .createListObjectsRequest(key,
- delimiter,
- span),
- ACCEPT_ALL,
- acceptor,
- cachedFilesIterator,
- span)),
- collectTombstones ? tombstones : null);
+ return createLocatedFileStatusIterator(
Review comment:
😀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r782145091
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -422,7 +315,7 @@ public void checkBasicFileOperations() throws Throwable {
readonlyFS.getFileStatus(emptyDir);
// now look at a file; the outcome depends on the mode.
- accessDeniedIf(!guardedInAuthMode, () ->
+ accessDeniedIf(!false, () ->
Review comment:
fixedl and updated the comment above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r783839214
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -3821,23 +3481,14 @@ S3AFileStatus s3GetFileStatus(final Path path,
// by the time of listing so that the response includes some
// which have not.
- int listSize;
- if (tombstones == null) {
- // no tombstones so look for a marker and at least one child.
- listSize = 2;
- } else {
- // build a listing > tombstones. If the caller has many thousands
- // of tombstones this won't work properly, which is why pruning
- // of expired tombstones matters.
- listSize = Math.min(2 + tombstones.size(), Math.max(2, maxKeys));
- }
+ final int listSize = 2;
S3ListRequest request = createListObjectsRequest(dirKey, "/",
listSize);
// execute the request
S3ListResult listResult = listObjects(request,
getDurationTrackerFactory());
- if (listResult.hasPrefixesOrObjects(contextAccessors, tombstones)) {
+ if (listResult.hasPrefixesOrObjects(contextAccessors, null)) {
Review comment:
Probably
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-969126236
testing s3 london, all good.
removed ITestPartialRenamesDeletes test which was failing as the output of the processing of the multi object delete exception didn't match expectations. We don't use that feature in production any more, once the need to update the s3guard tables goes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-947660904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974846673
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 15s | | trunk passed |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 7s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 2m 27s | [/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html) | hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 23m 42s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 19s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 21m 28s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 21m 28s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 19m 5s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 19m 5s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 48s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/results-checkstyle-root.txt) | root: The patch generated 16 new + 71 unchanged - 83 fixed = 87 total (was 154) |
| -1 :x: | mvnsite | 0m 36s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 39s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 35s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 24m 58s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 22s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 35s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. |
| | | 220m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle |
| uname | Linux 9a1ea1fa3335 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d174cba2f86fc0fa04c495430bf675b2175ddabc |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/testReport/ |
| Max. process+thread count | 1261 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974899616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-975888432
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 21s | | trunk passed |
| +1 :green_heart: | compile | 25m 18s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 9s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 33s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 8s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 35s | | the patch passed |
| +1 :green_heart: | compile | 23m 51s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 23m 51s | [/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 4 new + 1883 unchanged - 8 fixed = 1887 total (was 1891) |
| +1 :green_heart: | compile | 21m 21s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 21m 21s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 4 new + 1758 unchanged - 8 fixed = 1762 total (was 1766) |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 3m 49s | | root: The patch generated 0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 4m 29s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 7s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 32s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 1m 49s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 52s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 48s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +0 :ok: | spotbugs | 0m 40s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 21m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 39s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 18m 59s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 26s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 22s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not generate ASF License warnings. |
| | | 251m 2s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 8924d58c02b5 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1490e29ed8788d206c0523a920ab2722b9e22a45 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974862729
spotbugs is #3630;
compilation error is from the patch chain not deleting a file. squashing to see what that does
checkstyles are all valid; fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974846673
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 15s | | trunk passed |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 7s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 2m 27s | [/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html) | hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 23m 42s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 19s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 21m 28s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 21m 28s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 19m 5s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 19m 5s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 48s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/results-checkstyle-root.txt) | root: The patch generated 16 new + 71 unchanged - 83 fixed = 87 total (was 154) |
| -1 :x: | mvnsite | 0m 36s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 39s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 35s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 24m 58s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 22s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 35s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. |
| | | 220m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle |
| uname | Linux 9a1ea1fa3335 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d174cba2f86fc0fa04c495430bf675b2175ddabc |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/testReport/ |
| Max. process+thread count | 1261 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-976828074
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 41s | | trunk passed |
| +1 :green_heart: | compile | 28m 56s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 19s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 54s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 39s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 25m 0s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 52s | | the patch passed |
| +1 :green_heart: | compile | 29m 21s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 29m 21s | [/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 4 new + 1883 unchanged - 8 fixed = 1887 total (was 1891) |
| +1 :green_heart: | compile | 25m 36s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 25m 36s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 4 new + 1758 unchanged - 8 fixed = 1762 total (was 1766) |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 20s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 71 unchanged - 83 fixed = 74 total (was 154) |
| +1 :green_heart: | mvnsite | 4m 31s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 29s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 2m 4s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 50s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 47s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +0 :ok: | spotbugs | 0m 36s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 25m 2s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 20m 5s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 26s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 24s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not generate ASF License warnings. |
| | | 278m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux bb6ef3eaedd0 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 85eb597e525b811792735ad2fe9464f95a16fde9 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/testReport/ |
| Max. process+thread count | 1250 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-962095446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743649221
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentS3Object.java
##########
@@ -173,7 +173,7 @@ private void skipFailpoint(long len) throws IOException {
}
private boolean shouldInjectFailure(String key) {
- if (policy.shouldDelay(key) &&
+ if (false &&
readFailureCounter < MAX_READ_FAILURES) {
readFailureCounter++;
return true;
Review comment:
i've reinstated the whole ability to go inconsistent, in case anyone really wants it.
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -3587,116 +3399,13 @@ S3AFileStatus innerGetFileStatus(final Path f,
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({}); needEmptyDirectory={}",
path, key, needEmptyDirectoryFlag);
+ // there was no entry in S3Guard
+ // retrieve the data and update the metadata store in the process.
Review comment:
yeah, updated javadocs too. having inner and then s3 file status calls is surplus now; let's see about merging
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "",
+or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers, for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile, overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB: path names
-and summary details of objects —the actual data is stored in S3, so billed at S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests, combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in `core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0 --write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and `fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
-```
-hadoop s3guard import
-
-import [OPTIONS] [s3a://PATH]
- import metadata from existing S3 data
-
-Common options:
- -authoritative - Mark imported directory data as authoritative.
- -verbose - Verbose Output.
- -meta URL - Metadata repository details (implementation-specific)
-
-Amazon DynamoDB-specific options:
- -region REGION - Service region for connections
-
- URLs for Amazon DynamoDB are of the form dynamodb://TABLE_NAME.
- Specifying both the -region option and an S3A path
- is not supported.
-```
-
-Example
-
-Import all files and directories in a bucket into the S3Guard table.
-
-```bash
-hadoop s3guard import s3a://ireland-1
-```
-
-Import a directory tree, marking directories as authoritative.
-
-```bash
-hadoop s3guard import -authoritative -verbose s3a://ireland-1/fork-0008
-
-2020-01-03 12:05:18,321 [main] INFO - Metadata store DynamoDBMetadataStore{region=eu-west-1,
- tableName=s3guard-metadata, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/s3guard-metadata} is initialized.
-2020-01-03 12:05:18,324 [main] INFO - Starting: Importing s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,324 [main] INFO - Importing directory s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,537 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-false
-2020-01-03 12:05:18,630 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-true
-2020-01-03 12:05:19,142 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-0
-2020-01-03 12:05:19,191 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-1
-2020-01-03 12:05:19,240 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-0
-2020-01-03 12:05:19,289 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-1
-2020-01-03 12:05:19,314 [main] INFO - Updated S3Guard with 0 files and 6 directory entries
-2020-01-03 12:05:19,315 [main] INFO - Marking directory tree s3a://ireland-1/fork-0008 as authoritative
-2020-01-03 12:05:19,342 [main] INFO - Importing s3a://ireland-1/fork-0008: duration 0:01.018s
-Inserted 6 items into Metadata Store
-```
-
-### Compare a S3Guard table and the S3 Store: `s3guard diff`
-
-```bash
-hadoop s3guard diff [-meta URI] s3a://BUCKET
-```
-
-Lists discrepancies between a metadata store and bucket. Note that depending on
-how S3Guard is used, certain discrepancies are to be expected.
-
-Example
-
-```bash
-hadoop s3guard diff s3a://ireland-1
```
Review comment:
did a big review/cull of the doc again
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "",
+or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers, for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile, overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB: path names
-and summary details of objects —the actual data is stored in S3, so billed at S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests, combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in `core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0 --write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and `fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
Review comment:
cut too much; fixed
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -437,7 +330,7 @@ public void checkBasicFileOperations() throws Throwable {
// This means that permissions on the file do not get checked.
// See: HADOOP-16464.
Optional<FSDataInputStream> optIn = accessDeniedIf(
- !guardedInAuthMode, () -> readonlyFS.open(emptyFile));
+ !false, () -> readonlyFS.open(emptyFile));
Review comment:
removed the conditional nature entirely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014716071
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 110 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 33s | | trunk passed |
| +1 :green_heart: | compile | 24m 25s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 58s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 32s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 52s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 23s | | the patch passed |
| +1 :green_heart: | compile | 23m 43s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 43s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/19/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51 fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 20m 51s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 51s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/19/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688 unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 50s | | root: The patch generated 0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 44s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 45s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 39s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 41s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 37s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 24m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | | hadoop-project in the patch passed. |
| -1 :x: | unit | 18m 1s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/19/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 18s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 21s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 246m 40s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ipc.TestIPC |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 26ad7659d8e0 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b7fed1188783e2ef554a89fc756d94e07cf9f9b5 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/19/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/19/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014789921
done. will do the backporting next
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010549488
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 108 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 1s | | trunk passed |
| +1 :green_heart: | compile | 24m 35s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 41s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 49s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 36s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 22m 10s | [/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javac | 22m 10s | [/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 19m 44s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 19m 44s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 16s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/results-checkstyle-root.txt) | root: The patch generated 1 new + 71 unchanged - 84 fixed = 72 total (was 155) |
| -1 :x: | mvnsite | 0m 38s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 42s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 38s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 37s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from spotbugs |
| -1 :x: | spotbugs | 0m 37s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 24m 29s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 27s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 17m 47s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 37s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | unit | 2m 19s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 242m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 0c7b2b532db8 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / eaeb66df8fc75df15f6ae63078c8b0d217976592 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/testReport/ |
| Max. process+thread count | 2331 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743650731
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -3587,116 +3399,13 @@ S3AFileStatus innerGetFileStatus(final Path f,
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({}); needEmptyDirectory={}",
path, key, needEmptyDirectoryFlag);
+ // there was no entry in S3Guard
+ // retrieve the data and update the metadata store in the process.
Review comment:
yeah, updated javadocs too. having inner and then s3 file status calls is surplus now; let's see about merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743649221
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentS3Object.java
##########
@@ -173,7 +173,7 @@ private void skipFailpoint(long len) throws IOException {
}
private boolean shouldInjectFailure(String key) {
- if (policy.shouldDelay(key) &&
+ if (false &&
readFailureCounter < MAX_READ_FAILURES) {
readFailureCounter++;
return true;
Review comment:
i've reinstated the whole ability to go inconsistent, in case anyone really wants it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743692552
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -437,7 +330,7 @@ public void checkBasicFileOperations() throws Throwable {
// This means that permissions on the file do not get checked.
// See: HADOOP-16464.
Optional<FSDataInputStream> optIn = accessDeniedIf(
- !guardedInAuthMode, () -> readonlyFS.open(emptyFile));
+ !false, () -> readonlyFS.open(emptyFile));
Review comment:
removed the conditional nature entirely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974899616
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 34s | | trunk passed |
| +1 :green_heart: | compile | 23m 38s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 10s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 14s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 2m 27s | [/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html) | hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 23m 48s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 22m 50s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 22m 50s | [/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1888 unchanged - 5 fixed = 1889 total (was 1893) |
| +1 :green_heart: | compile | 20m 8s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 20m 8s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1764 unchanged - 5 fixed = 1765 total (was 1769) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 52s | | root: The patch generated 0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 36s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| -1 :x: | spotbugs | 1m 24s | [/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 24m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 21s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 44s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. |
| | | 226m 51s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Dead store to isDir in org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(String, long, String, String) At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(String, long, String, String) At S3AFileSystem.java:[line 4018] |
| | Unused field:S3AMultipartUploader.java |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle |
| uname | Linux a7c8887dc631 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / adaa03ea39792c0cf2aa84c145a46a8091b507e3 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/testReport/ |
| Max. process+thread count | 2878 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014438843
thanks. realised there's one thing left -an entry in troubleshooting about the new error message. building a release, generating the real output and adding. this is doc only change so shouldn't force a new review.
time to "knife the baby" , as bill gates said to steve jobs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010386405
hboss is just using the config options to switch to local s3guard in tests. we leave the options alone
but: should we treat asking for local as a no op? because they call it to get a consistent s3 store...there's no risk of ddb tables corrupting the view of other processes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010898339
test run s3 london. `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=delete -Dscale`
one failure from DT tests as I'd turned session tokens on there, otherwise all good. filing a jira to get the failing test to clear more options
```
[ERROR] Failures:
[ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:303->AbstractDelegationIT.lookupToken:70->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 Kind of token Kind: S3ADelegationToken/Session, Service: s3a://stevel-london, Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; uri=s3a://stevel-london; timestamp=1641943367014; renewer=stevel; encryption=SSE-KMS; df39af66-6a55-4d84-9ce0-7be719f58087; Created on stevel-mbp1376/127.0.0.1 at time 2022-01-11T23:22:46.805Z.}; session credentials, expiry 2022-01-12T11:22:47Z; (valid)) expected:<S3ADelegationToken/Full> but was:<S3ADelegationToken/Session>
[ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:303->AbstractDelegationIT.lookupToken:70->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 Kind of token Kind: S3ADelegationToken/Session, Service: s3a://stevel-london, Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; uri=s3a://stevel-london; timestamp=1641943383624; renewer=stevel; encryption=SSE-KMS; fcf63971-7249-4339-92c1-89a508fee7d2; Created on stevel-mbp1376/127.0.0.1 at time 2022-01-11T23:23:03.247Z.}; session credentials, expiry 2022-01-12T11:23:03Z; (valid)) expected:<S3ADelegationToken/Role> but was:<S3ADelegationToken/Session>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011292027
and warnings
```
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitOperations.java:436:30:[deprecation] METADATASTORE_AUTHORITATIVE in Constants has been deprecated
```
it has been adding some metastore info to the job commit. i will cut it all out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011290633
hadoop common failure is testIpc -covered in HADOOP-18046 and unrelated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011534353
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 109 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 59s | | trunk passed |
| +1 :green_heart: | compile | 24m 17s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 54s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 32s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 24m 22s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | | the patch passed |
| +1 :green_heart: | compile | 23m 29s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 29s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 5 new + 1813 unchanged - 51 fixed = 1818 total (was 1864) |
| +1 :green_heart: | compile | 20m 46s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 46s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 5 new + 1688 unchanged - 51 fixed = 1693 total (was 1739) |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/blanks-eol.txt) | The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 49s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/results-checkstyle-root.txt) | root: The patch generated 8 new + 71 unchanged - 84 fixed = 79 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 44s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 46s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 39s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 37s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 30s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 23m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | | hadoop-project in the patch passed. |
| -1 :x: | unit | 17m 44s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 18s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 20s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 247m 30s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ipc.TestIPC |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 0fa577c0bf03 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3a3ed08f843074de78c278fbc4f70c7b44c8b723 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/testReport/ |
| Max. process+thread count | 2144 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012330131
mistakenly got the test changes for #3864 in this line; will remove and force the branch up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012297514
latest build tested against s3 london, `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep -Dscale`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r782538228
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -4284,31 +3929,8 @@ private CopyResult copyFile(String srcKey, String dstKey, long size,
String action = "copyFile(" + srcKey + ", " + dstKey + ")";
Invoker readInvoker = readContext.getReadInvoker();
- ObjectMetadata srcom;
- try {
- srcom = once(action, srcKey,
- () ->
- getObjectMetadata(srcKey, changeTracker, readInvoker, "copy"));
- } catch (FileNotFoundException e) {
- // if rename fails at this point it means that the expected file was not
- // found.
- // The cause is believed to always be one of
- // - File was deleted since LIST/S3Guard metastore.list.() knew of it.
- // - S3Guard is asking for a specific version and it's been removed by
- // lifecycle rules.
- // - there's a 404 cached in the S3 load balancers.
- LOG.debug("getObjectMetadata({}) failed to find an expected file",
- srcKey, e);
- // We create an exception, but the text depends on the S3Guard state
- String message = hasMetadataStore()
Review comment:
this is during a rename where the list call found the file, now we call copy and it's not there.
the causes used to be
1. s3guard inconsistent
2. unguarded and 404 caching making the file not visible
3. something else deleted the file between list and copy
only #3 can happen now, so i was just leaving as it.
but I could revert, as we may want to differentiate the failure.
will revert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r783839500
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RenameOperation.java
##########
@@ -724,4 +608,22 @@ private String maybeAddTrailingSlash(String key) {
}
}
+ /**
+ * Convert a passed in exception (expected to be an IOE or AWS exception)
+ * into an IOException.
+ * @param ex exception caught
+ * @return the exception to throw in the failure handler.
+ */
+ protected IOException convertToIOException(final Exception ex) {
+ if (ex instanceof IOException) {
+ return (IOException) ex;
Review comment:
so we can return it as an IOE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011191145
`[ERROR] Tests run: 15, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 31.898 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool
[ERROR] testLandsatBucketRequireUnencrypted(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool) Time elapsed: 2.139 s <<< ERROR!
46: Bucket s3a://landsat-pds: required encryption is none but actual encryption is CSE-KMS
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.exitException(S3GuardTool.java:914)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.badState(S3GuardTool.java:880)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:509)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:282)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:962)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardToolTestHelper.runS3GuardCommand(S3GuardToolTestHelper.java:147)
at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.run(AbstractS3GuardToolTestBase.java:124)
at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool.testLandsatBucketRequireUnencrypted(ITestS3GuardTool.java:73)`
Seeing this failure in my setup while raw S3 tests. Shouldn't this test be removed ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r754893451
##########
File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
##########
@@ -343,7 +343,7 @@ stores pretend that they are a FileSystem, a FileSystem with the same
features and operations as HDFS. This is —ultimately—a pretence:
they have different characteristics and occasionally the illusion fails.
-1. **Consistency**. Object stores are generally *Eventually Consistent*: it
+1. **Consistency**. Object mqy be *Eventually Consistent*: it
Review comment:
nit : typo mqy -> may
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java
##########
@@ -279,82 +205,19 @@ private ObjectListingIterator createObjectListingIterator(
LOG.debug("Requesting all entries under {} with delimiter '{}'",
key, delimiter);
}
- final RemoteIterator<S3AFileStatus> cachedFilesIterator;
- final Set<Path> tombstones;
- boolean allowAuthoritative = listingOperationCallbacks
- .allowAuthoritative(path);
- if (recursive) {
- final PathMetadata pm = getStoreContext()
- .getMetadataStore()
- .get(path, true);
- if (pm != null) {
- if (pm.isDeleted()) {
- OffsetDateTime deletedAt = OffsetDateTime
- .ofInstant(Instant.ofEpochMilli(
- pm.getFileStatus().getModificationTime()),
- ZoneOffset.UTC);
- throw new FileNotFoundException("Path " + path + " is recorded as " +
- "deleted by S3Guard at " + deletedAt);
- }
- }
- MetadataStoreListFilesIterator metadataStoreListFilesIterator =
- new MetadataStoreListFilesIterator(
- getStoreContext().getMetadataStore(),
- pm,
- allowAuthoritative);
- tombstones = metadataStoreListFilesIterator.listTombstones();
- // if all of the below is true
- // - authoritative access is allowed for this metadatastore
- // for this directory,
- // - all the directory listings are authoritative on the client
- // - the caller does not force non-authoritative access
- // return the listing without any further s3 access
- if (!forceNonAuthoritativeMS &&
- allowAuthoritative &&
- metadataStoreListFilesIterator.isRecursivelyAuthoritative()) {
- S3AFileStatus[] statuses = S3AUtils.iteratorToStatuses(
- metadataStoreListFilesIterator, tombstones);
- cachedFilesIterator = createProvidedFileStatusIterator(
- statuses, ACCEPT_ALL, acceptor);
- return createLocatedFileStatusIterator(cachedFilesIterator);
- }
- cachedFilesIterator = metadataStoreListFilesIterator;
- } else {
- DirListingMetadata meta =
- S3Guard.listChildrenWithTtl(
- getStoreContext().getMetadataStore(),
- path,
- listingOperationCallbacks.getUpdatedTtlTimeProvider(),
- allowAuthoritative);
- if (meta != null) {
- tombstones = meta.listTombstones();
- } else {
- tombstones = null;
- }
- cachedFilesIterator = createProvidedFileStatusIterator(
- S3Guard.dirMetaToStatuses(meta), ACCEPT_ALL, acceptor);
- if (allowAuthoritative && meta != null && meta.isAuthoritative()) {
- // metadata listing is authoritative, so return it directly
- return createLocatedFileStatusIterator(cachedFilesIterator);
- }
- }
- return createTombstoneReconcilingIterator(
- createLocatedFileStatusIterator(
- createFileStatusListingIterator(path,
- listingOperationCallbacks
- .createListObjectsRequest(key,
- delimiter,
- span),
- ACCEPT_ALL,
- acceptor,
- cachedFilesIterator,
- span)),
- collectTombstones ? tombstones : null);
+ return createLocatedFileStatusIterator(
Review comment:
Wow. Listing is super simple now. 💯
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -4284,31 +3929,8 @@ private CopyResult copyFile(String srcKey, String dstKey, long size,
String action = "copyFile(" + srcKey + ", " + dstKey + ")";
Invoker readInvoker = readContext.getReadInvoker();
- ObjectMetadata srcom;
- try {
- srcom = once(action, srcKey,
- () ->
- getObjectMetadata(srcKey, changeTracker, readInvoker, "copy"));
- } catch (FileNotFoundException e) {
- // if rename fails at this point it means that the expected file was not
- // found.
- // The cause is believed to always be one of
- // - File was deleted since LIST/S3Guard metastore.list.() knew of it.
- // - S3Guard is asking for a specific version and it's been removed by
- // lifecycle rules.
- // - there's a 404 cached in the S3 load balancers.
- LOG.debug("getObjectMetadata({}) failed to find an expected file",
- srcKey, e);
- // We create an exception, but the text depends on the S3Guard state
- String message = hasMetadataStore()
Review comment:
nit: Won't this make the exception returned to client backward incompatible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r753787918
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -422,7 +315,7 @@ public void checkBasicFileOperations() throws Throwable {
readonlyFS.getFileStatus(emptyDir);
// now look at a file; the outcome depends on the mode.
- accessDeniedIf(!guardedInAuthMode, () ->
+ accessDeniedIf(!false, () ->
Review comment:
came from me fixing the calls to be true/false and then inlining; can make clause non-conditional
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java
##########
@@ -60,12 +55,13 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
/**
* A wrapper around {@link com.amazonaws.services.s3.AmazonS3} that injects
- * inconsistency and/or errors. Used for testing S3Guard.
- * Currently only delays listing visibility, not affecting GET.
+ * failures.
+ * It used to also inject inconsistency, but this was removed with S3Guard;
+ * what is retained is the ability to throttle AWS operations and for the
+ * input stream to be inconsistent.
*/
@InterfaceAudience.Private
@InterfaceStability.Unstable
Review comment:
keeping failure injection as it can set read failures and throttling, but cutting paths for delayed listing, the settings/probes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r753787918
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -422,7 +315,7 @@ public void checkBasicFileOperations() throws Throwable {
readonlyFS.getFileStatus(emptyDir);
// now look at a file; the outcome depends on the mode.
- accessDeniedIf(!guardedInAuthMode, () ->
+ accessDeniedIf(!false, () ->
Review comment:
came from me fixing the calls to be true/false and then inlining; can make clause non-conditional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-977215558
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 9s | | trunk passed |
| +1 :green_heart: | compile | 23m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 8s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 44s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 19s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 16s | | the patch passed |
| +1 :green_heart: | compile | 22m 54s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 22m 54s | [/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1888 unchanged - 5 fixed = 1889 total (was 1893) |
| +1 :green_heart: | compile | 20m 13s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 20m 13s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1764 unchanged - 5 fixed = 1765 total (was 1769) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 51s | | root: The patch generated 0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 3m 38s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 41s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 1m 37s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 24m 4s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 17m 56s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 39s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 10s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. |
| | | 242m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 2466915c94e0 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c347406b54a5fba45940ab93b87ffdd79a70281e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/testReport/ |
| Max. process+thread count | 1260 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-938723368
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 40 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 17s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 45s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 34s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 46s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 46s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 34s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 0m 34s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 23s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 16 new + 47 unchanged - 34 fixed = 63 total (was 81) |
| -1 :x: | mvnsite | 0m 33s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 36s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 26m 33s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 36s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 108m 43s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux 70dac4eed851 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 01fd32e19a59fe7bc2e0cc72002a39eacd42d3ef |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/testReport/ |
| Max. process+thread count | 622 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743649221
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentS3Object.java
##########
@@ -173,7 +173,7 @@ private void skipFailpoint(long len) throws IOException {
}
private boolean shouldInjectFailure(String key) {
- if (policy.shouldDelay(key) &&
+ if (false &&
readFailureCounter < MAX_READ_FAILURES) {
readFailureCounter++;
return true;
Review comment:
i've reinstated the whole ability to go inconsistent, in case anyone really wants it.
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -3587,116 +3399,13 @@ S3AFileStatus innerGetFileStatus(final Path f,
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({}); needEmptyDirectory={}",
path, key, needEmptyDirectoryFlag);
+ // there was no entry in S3Guard
+ // retrieve the data and update the metadata store in the process.
Review comment:
yeah, updated javadocs too. having inner and then s3 file status calls is surplus now; let's see about merging
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "",
+or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers, for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile, overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB: path names
-and summary details of objects —the actual data is stored in S3, so billed at S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests, combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in `core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0 --write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and `fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
-```
-hadoop s3guard import
-
-import [OPTIONS] [s3a://PATH]
- import metadata from existing S3 data
-
-Common options:
- -authoritative - Mark imported directory data as authoritative.
- -verbose - Verbose Output.
- -meta URL - Metadata repository details (implementation-specific)
-
-Amazon DynamoDB-specific options:
- -region REGION - Service region for connections
-
- URLs for Amazon DynamoDB are of the form dynamodb://TABLE_NAME.
- Specifying both the -region option and an S3A path
- is not supported.
-```
-
-Example
-
-Import all files and directories in a bucket into the S3Guard table.
-
-```bash
-hadoop s3guard import s3a://ireland-1
-```
-
-Import a directory tree, marking directories as authoritative.
-
-```bash
-hadoop s3guard import -authoritative -verbose s3a://ireland-1/fork-0008
-
-2020-01-03 12:05:18,321 [main] INFO - Metadata store DynamoDBMetadataStore{region=eu-west-1,
- tableName=s3guard-metadata, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/s3guard-metadata} is initialized.
-2020-01-03 12:05:18,324 [main] INFO - Starting: Importing s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,324 [main] INFO - Importing directory s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,537 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-false
-2020-01-03 12:05:18,630 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-true
-2020-01-03 12:05:19,142 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-0
-2020-01-03 12:05:19,191 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-1
-2020-01-03 12:05:19,240 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-0
-2020-01-03 12:05:19,289 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-1
-2020-01-03 12:05:19,314 [main] INFO - Updated S3Guard with 0 files and 6 directory entries
-2020-01-03 12:05:19,315 [main] INFO - Marking directory tree s3a://ireland-1/fork-0008 as authoritative
-2020-01-03 12:05:19,342 [main] INFO - Importing s3a://ireland-1/fork-0008: duration 0:01.018s
-Inserted 6 items into Metadata Store
-```
-
-### Compare a S3Guard table and the S3 Store: `s3guard diff`
-
-```bash
-hadoop s3guard diff [-meta URI] s3a://BUCKET
-```
-
-Lists discrepancies between a metadata store and bucket. Note that depending on
-how S3Guard is used, certain discrepancies are to be expected.
-
-Example
-
-```bash
-hadoop s3guard diff s3a://ireland-1
```
Review comment:
did a big review/cull of the doc again
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "",
+or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers, for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile, overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB: path names
-and summary details of objects —the actual data is stored in S3, so billed at S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests, combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request" capacity.
- Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in `core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0 --write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and `fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
Review comment:
cut too much; fixed
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -437,7 +330,7 @@ public void checkBasicFileOperations() throws Throwable {
// This means that permissions on the file do not get checked.
// See: HADOOP-16464.
Optional<FSDataInputStream> optIn = accessDeniedIf(
- !guardedInAuthMode, () -> readonlyFS.open(emptyFile));
+ !false, () -> readonlyFS.open(emptyFile));
Review comment:
removed the conditional nature entirely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-976828074
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 41s | | trunk passed |
| +1 :green_heart: | compile | 28m 56s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 19s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 54s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 39s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 25m 0s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 52s | | the patch passed |
| +1 :green_heart: | compile | 29m 21s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 29m 21s | [/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 4 new + 1883 unchanged - 8 fixed = 1887 total (was 1891) |
| +1 :green_heart: | compile | 25m 36s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 25m 36s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 4 new + 1758 unchanged - 8 fixed = 1762 total (was 1766) |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 20s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 71 unchanged - 83 fixed = 74 total (was 154) |
| +1 :green_heart: | mvnsite | 4m 31s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 29s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 2m 4s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 50s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 47s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +0 :ok: | spotbugs | 0m 36s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 25m 2s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 20m 5s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 26s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 24s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not generate ASF License warnings. |
| | | 278m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux bb6ef3eaedd0 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 85eb597e525b811792735ad2fe9464f95a16fde9 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/testReport/ |
| Max. process+thread count | 1250 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-975468562
minor spotbugs to fix up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974862729
spotbugs is #3630;
compilation error is from the patch chain not deleting a file. squashing to see what that does
checkstyles are all valid; fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-976586826
HBoss contains references to local metastore & ability to set it in
hadoop-testutils/src/main/java/org/apache/hadoop/hbase/oss/EmbeddedS3.java
maybe we should make that a no-op. or at least fix up hboss to never use it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012537514
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 111 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 13s | | trunk passed |
| +1 :green_heart: | compile | 28m 49s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 24m 5s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 49s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 24s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 27m 25s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 41s | | the patch passed |
| +1 :green_heart: | compile | 28m 26s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 28m 26s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51 fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 25m 24s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 25m 24s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688 unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 23s | | root: The patch generated 0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 4m 55s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 27s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 2m 4s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 49s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 46s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 31s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 28m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-project in the patch passed. |
| -1 :x: | unit | 19m 9s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 36s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 32s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not generate ASF License warnings. |
| | | 285m 15s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ipc.TestIPC |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 8d38908922d2 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a94e0e103fb3a6232cfff03e12671f12e89c123c |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/testReport/ |
| Max. process+thread count | 2932 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014745441
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 110 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 46s | | trunk passed |
| +1 :green_heart: | compile | 22m 21s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 35s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 18s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 57s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 42s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 20m 53s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 21s | | the patch passed |
| +1 :green_heart: | compile | 21m 32s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 32s | | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 0 new + 1816 unchanged - 50 fixed = 1816 total (was 1866) |
| +1 :green_heart: | compile | 19m 32s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 32s | | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 1692 unchanged - 50 fixed = 1692 total (was 1742) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 34s | | root: The patch generated 0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 17s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 35s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 46s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 48s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 46s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 38s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 21m 13s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 18m 11s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 19s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 24s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 235m 11s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/20/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux e855d7f7f82b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4cfc07eb195d65aa9f15dbdcdebc8d2caf0340a2 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/20/testReport/ |
| Max. process+thread count | 1266 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/20/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011428802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r782538228
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -4284,31 +3929,8 @@ private CopyResult copyFile(String srcKey, String dstKey, long size,
String action = "copyFile(" + srcKey + ", " + dstKey + ")";
Invoker readInvoker = readContext.getReadInvoker();
- ObjectMetadata srcom;
- try {
- srcom = once(action, srcKey,
- () ->
- getObjectMetadata(srcKey, changeTracker, readInvoker, "copy"));
- } catch (FileNotFoundException e) {
- // if rename fails at this point it means that the expected file was not
- // found.
- // The cause is believed to always be one of
- // - File was deleted since LIST/S3Guard metastore.list.() knew of it.
- // - S3Guard is asking for a specific version and it's been removed by
- // lifecycle rules.
- // - there's a 404 cached in the S3 load balancers.
- LOG.debug("getObjectMetadata({}) failed to find an expected file",
- srcKey, e);
- // We create an exception, but the text depends on the S3Guard state
- String message = hasMetadataStore()
Review comment:
we always generated different messages when s3guard was on/off, didn;t we?
now you only get the unguarded text
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012537514
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 111 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 13s | | trunk passed |
| +1 :green_heart: | compile | 28m 49s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 24m 5s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 49s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 24s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 27m 25s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 41s | | the patch passed |
| +1 :green_heart: | compile | 28m 26s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 28m 26s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51 fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 25m 24s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 25m 24s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688 unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 23s | | root: The patch generated 0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 4m 55s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 27s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 2m 4s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 49s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 46s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 31s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 28m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-project in the patch passed. |
| -1 :x: | unit | 19m 9s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 36s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 32s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not generate ASF License warnings. |
| | | 285m 15s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ipc.TestIPC |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 8d38908922d2 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a94e0e103fb3a6232cfff03e12671f12e89c123c |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/testReport/ |
| Max. process+thread count | 2932 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-964130727
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 106 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 48s | | trunk passed |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 0s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 23s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 23s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 22s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 0m 22s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 20 new + 71 unchanged - 83 fixed = 91 total (was 154) |
| -1 :x: | mvnsite | 0m 24s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 26s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 22s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 20m 34s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 26s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 82m 24s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle markdownlint |
| uname | Linux 20bce25a2c05 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ce7e4ac5b6c7f9dea43d3b3ac47df23687963017 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/testReport/ |
| Max. process+thread count | 545 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-962095446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
mehakmeet commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r752291332
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java
##########
@@ -60,12 +55,13 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
/**
* A wrapper around {@link com.amazonaws.services.s3.AmazonS3} that injects
- * inconsistency and/or errors. Used for testing S3Guard.
- * Currently only delays listing visibility, not affecting GET.
+ * failures.
+ * It used to also inject inconsistency, but this was removed with S3Guard;
+ * what is retained is the ability to throttle AWS operations and for the
+ * input stream to be inconsistent.
*/
@InterfaceAudience.Private
@InterfaceStability.Unstable
Review comment:
Should we remove some methods like `setFailureInjectionPolicy()` and `clearInconsistency()` which are explicitly trying to create S3 client for "s3guard"
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFailureHandling.java
##########
@@ -171,14 +143,10 @@ public void testMultiObjectDeleteMissingEntriesSucceeds() throws Throwable {
List<DeleteObjectsRequest.KeyVersion> keys = keysToDelete(
Review comment:
Ref of `TestPartialDeleteFailures` and `MultiObjectDeleteSupport` which are removed now.
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -1014,6 +931,7 @@ private Constants() {
"2s";
Review comment:
Should mark S3GUARD_CONSISTENCY_RETRY_INTERVAL configs as Deprecated as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974088101
in RenameOperation & DeleteOperatoin, move off guava preconditions at the same time; fix imports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-947660904
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 94 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 15s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 17s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 0m 41s | [/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 12 new + 16 unchanged - 5 fixed = 28 total (was 21) |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 0m 33s | [/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 11 new + 16 unchanged - 5 fixed = 27 total (was 21) |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 19s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 29 new + 54 unchanged - 81 fixed = 83 total (was 135) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 26s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 41 unchanged - 21 fixed = 42 total (was 62) |
| -1 :x: | spotbugs | 1m 16s | [/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 25m 28s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 50s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 105m 1s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Dead store to isDir in org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(String, long, String, String) At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(String, long, String, String) At S3AFileSystem.java:[line 4060] |
| | Unused field:S3AMultipartUploader.java |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle markdownlint |
| uname | Linux de7a64d6cbd5 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2d6644ac4edfa9e0b47e2e049cd3b03e4e46cfe1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/testReport/ |
| Max. process+thread count | 569 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-975888432
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 21s | | trunk passed |
| +1 :green_heart: | compile | 25m 18s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 9s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 33s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 8s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 35s | | the patch passed |
| +1 :green_heart: | compile | 23m 51s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 23m 51s | [/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 4 new + 1883 unchanged - 8 fixed = 1887 total (was 1891) |
| +1 :green_heart: | compile | 21m 21s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 21m 21s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 4 new + 1758 unchanged - 8 fixed = 1762 total (was 1766) |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 3m 49s | | root: The patch generated 0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 4m 29s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 7s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 32s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 1m 49s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 52s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 48s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +0 :ok: | spotbugs | 0m 40s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 21m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 39s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 18m 59s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 26s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 22s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not generate ASF License warnings. |
| | | 251m 2s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 8924d58c02b5 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1490e29ed8788d206c0523a920ab2722b9e22a45 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] harshavardhana commented on a change in pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
harshavardhana commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r755876603
##########
File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
##########
@@ -343,7 +343,7 @@ stores pretend that they are a FileSystem, a FileSystem with the same
features and operations as HDFS. This is —ultimately—a pretence:
they have different characteristics and occasionally the illusion fails.
-1. **Consistency**. Object stores are generally *Eventually Consistent*: it
+1. **Consistency**. Object mqy be *Eventually Consistent*: it
Review comment:
```suggestion
1. **Consistency**. Object may be *Eventually Consistent*: it
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-947660904
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 94 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 15s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 17s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 0m 41s | [/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 12 new + 16 unchanged - 5 fixed = 28 total (was 21) |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 0m 33s | [/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 11 new + 16 unchanged - 5 fixed = 27 total (was 21) |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 19s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 29 new + 54 unchanged - 81 fixed = 83 total (was 135) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 26s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 41 unchanged - 21 fixed = 42 total (was 62) |
| -1 :x: | spotbugs | 1m 16s | [/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 25m 28s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 50s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 105m 1s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Dead store to isDir in org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(String, long, String, String) At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(String, long, String, String) At S3AFileSystem.java:[line 4060] |
| | Unused field:S3AMultipartUploader.java |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle markdownlint |
| uname | Linux de7a64d6cbd5 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2d6644ac4edfa9e0b47e2e049cd3b03e4e46cfe1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/testReport/ |
| Max. process+thread count | 569 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010549488
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 108 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 1s | | trunk passed |
| +1 :green_heart: | compile | 24m 35s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 41s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 49s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 36s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 22m 10s | [/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javac | 22m 10s | [/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 19m 44s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 19m 44s | [/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 16s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/results-checkstyle-root.txt) | root: The patch generated 1 new + 71 unchanged - 84 fixed = 72 total (was 155) |
| -1 :x: | mvnsite | 0m 38s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 42s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 38s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 37s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from spotbugs |
| -1 :x: | spotbugs | 0m 37s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 24m 29s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 27s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 17m 47s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 37s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | unit | 2m 19s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 242m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 0c7b2b532db8 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / eaeb66df8fc75df15f6ae63078c8b0d217976592 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/testReport/ |
| Max. process+thread count | 2331 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
mehakmeet commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010976232
+1 pending the yetus checks, tested on non-CSE and CSE test suite, everything went fine and getting the correct "not supported" error when turning s3guard on.
One suggestion would be maybe referencing this Jira ticket in `s3guard.md` where we are explaining that it was removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011225626
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 108 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 42s | | trunk passed |
| +1 :green_heart: | compile | 24m 16s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 34s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 49s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 19s | | the patch passed |
| +1 :green_heart: | compile | 28m 14s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 28m 14s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 4 new + 1813 unchanged - 51 fixed = 1817 total (was 1864) |
| +1 :green_heart: | compile | 22m 36s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 22m 36s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 4 new + 1688 unchanged - 51 fixed = 1692 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 54s | | root: The patch generated 0 new + 71 unchanged - 84 fixed = 71 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 45s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 38s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 39s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 31s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 23m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | | hadoop-project in the patch passed. |
| -1 :x: | unit | 18m 9s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 18s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 20s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. |
| | | 251m 51s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ipc.TestIPC |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux dece1db4343d 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 005dcfab3004db8c679456731ff42911a0ef4550 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-977215558
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 107 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 9s | | trunk passed |
| +1 :green_heart: | compile | 23m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 8s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 44s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 19s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 16s | | the patch passed |
| +1 :green_heart: | compile | 22m 54s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 22m 54s | [/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1888 unchanged - 5 fixed = 1889 total (was 1893) |
| +1 :green_heart: | compile | 20m 13s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 20m 13s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1764 unchanged - 5 fixed = 1765 total (was 1769) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 51s | | root: The patch generated 0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 3m 38s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 41s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 1m 37s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 24m 4s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 17m 56s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 39s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 10s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. |
| | | 242m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 2466915c94e0 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c347406b54a5fba45940ab93b87ffdd79a70281e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/testReport/ |
| Max. process+thread count | 1260 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur edited a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
mukund-thakur edited a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011970010
> `[ERROR] Failures: [ERROR] ITestS3AMiscOperations.testEmptyFileChecksums:182->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums of empty files expected:<etag: "5334c7a47521faa37aaaadb3eda3cc7c"> but was:<etag: "37d542d4dade6ea73842e723b57b4c9b"> [ERROR] ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted:258->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums expected:<etag: "057988c6b2a0f57adf8c83c695059978"> but was:<etag: "7ecdd0d908e71fc933749af8612460d7"> [ERROR] Errors: `
>
> Also these failures not sure if they are related.
This is failing in trunk as well even after removing the encryption configs from the auth-keys. Although I have enabled the encryption as SSE-KMS in aws console.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012520838
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 110 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 65m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 5s | | trunk passed |
| +1 :green_heart: | compile | 24m 26s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 46s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 57s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | | the patch passed |
| +1 :green_heart: | compile | 23m 30s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 30s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/16/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51 fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 20m 43s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 43s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/16/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688 unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 45s | | root: The patch generated 0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 39s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 41s | | hadoop-common in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-azure in the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 24m 8s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 27s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 17m 45s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 29s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 5s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 297m 17s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3534 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux a2db65b34ec4 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ac185db64d6e8c78178fb52b46f0f7b5c92344ab |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/16/testReport/ |
| Max. process+thread count | 2972 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/16/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3534: HADOOP-17409. Remove s3guard from S3A module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011360617
@mukund-thakur sounds like your encryption setting settings are causing problems.
does this happen on other branches? maybe I should rebase this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org