You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/06/21 09:31:51 UTC
[GitHub] [hadoop] dannycjones opened a new pull request, #4478: HADOOP-18304. Improve user-facing S3A committers documentation
dannycjones opened a new pull request, #4478:
URL: https://github.com/apache/hadoop/pull/4478
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
As noted in the ticket, this PR attempts to improve the committer docs given a fresh pair of eyes from someone who has not worked with the committers before.
I've tried to ensure that the Table of Contents makes more sense too.
### How was this patch tested?
Reading :)
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mehakmeet merged pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
mehakmeet merged PR #4478:
URL: https://github.com/apache/hadoop/pull/4478
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1161774227
@ahmarsuhail - would you be able to review this week?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r904883423
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -530,18 +527,22 @@ performance.
### Enabling the committer
+Set the committer used by S3A's committer factory to `magic`:
+
```xml
Review Comment:
It's a good question. I do worry we duplicate this in a lot of places, not sure what is the best option.
I plan to leave it out of the scope of this patch though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1283734123
Thanks Mehakmeet! I've opened #5043 for backport to `branch-3.3`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r904893346
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -492,18 +484,19 @@ was written. With the policy of `append`, the new file would be added to
the existing set of files.
-### Notes
+### Notes on using Staging Committers
1. A deep partition tree can itself be a performance problem in S3 and the s3a client,
-or, more specifically. a problem with applications which use recursive directory tree
+or more specifically a problem with applications which use recursive directory tree
walks to work with data.
1. The outcome if you have more than one job trying simultaneously to write data
to the same destination with any policy other than "append" is undefined.
1. In the `append` operation, there is no check for conflict with file names.
-If, in the example above, the file `log-20170228.avro` already existed,
-it would be overridden. Set `fs.s3a.committer.staging.unique-filenames` to `true`
+If the file `log-20170228.avro` in the example above already existed, it would be overwritten.
+
+ Set `fs.s3a.committer.staging.unique-filenames` to `true`
Review Comment:
Using the indentation like this I believe allows you to put the sentence on a new line but still part of the previous point.
That being said, it is not obvious from the markdown and I cannot test the output HTML so I'll revert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1271440014
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 69m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 32s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 97m 3s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4478 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets markdownlint |
| uname | Linux 84475ee88fc4 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a56691d39a8548f519d1c9759b41d4fb24920a6d |
| Max. process+thread count | 533 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/8/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r904889810
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -650,7 +639,14 @@ Conflict management is left to the execution engine itself.
</property>
Review Comment:
Good call out, I had a look and I think that option should actually be just `fs.s3a.committer.uuid` - no `staging`. I'll update it.
The code in `AbstractS3ACommitter` checks for `fs.s3a.committer.uuid` and the Spark UUID before failing fast.
https://github.com/apache/hadoop/blob/e199da3fae1c82e87f88c8c50f6a96c6515e2edd/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L1341-L1360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r989873742
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -180,20 +180,20 @@ and restarting the job.
whose output is in the job attempt directory, *and only rerunning all uncommitted tasks*.
-This algorithm does not works safely or swiftly with AWS S3 storage because
-tenames go from being fast, atomic operations to slow operations which can fail partway through.
+This algorithm does not work safely or swiftly with AWS S3 storage because
+renames go from being fast, atomic operations to slow operations which can fail partway through.
This then is the problem which the S3A committers address:
-*How to safely and reliably commit work to Amazon S3 or compatible object store*
+>*How to safely and reliably commit work to Amazon S3 or compatible object store.*
Review Comment:
Fair point, it is not quoting anyone (or maybe Steve). I will let it join the previous sentence as I think that makes more sense.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -356,19 +351,20 @@ task commit.
However, it has extra requirements of the filesystem
-1. [Obsolete] It requires a consistent object store.
+1. The object store must be consistent.
1. The S3A client must be configured to recognize interactions
-with the magic directories and treat them specially.
+with the magic directories and treat them as a special case.
-Now that Amazon S3 is consistent, the magic committer is enabled by default.
+Now that [Amazon S3 is consistent](https://aws.amazon.com/s3/consistency/),
+the magic committer is enabled by default.
Review Comment:
Yes, I believe that's correct.
I believe what we're saying here is that S3A's "magic path rewriting" where it only stages the writes to `__magic/` directories is now enabled by default.
I will update this to be clearer, something like:
```suggestion
the magic directory path rewriting is enabled by default.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
mehakmeet commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1269937773
Yes @dannycjones, I'll check this by tomorrow morning IST.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1164395425
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| -1 :x: | shadedclient | 86m 56s | | branch has errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 52s | | the patch passed |
| -1 :x: | shadedclient | 5m 10s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 34s | | ASF License check generated no output? |
| | | 96m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4478 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets markdownlint |
| uname | Linux e68a2eb56e51 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cd600503542751a72b7b21028ef08b26e41e6580 |
| Max. process+thread count | 546 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/3/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
mehakmeet commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r989684378
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -356,19 +351,20 @@ task commit.
However, it has extra requirements of the filesystem
-1. [Obsolete] It requires a consistent object store.
+1. The object store must be consistent.
1. The S3A client must be configured to recognize interactions
-with the magic directories and treat them specially.
+with the magic directories and treat them as a special case.
-Now that Amazon S3 is consistent, the magic committer is enabled by default.
+Now that [Amazon S3 is consistent](https://aws.amazon.com/s3/consistency/),
+the magic committer is enabled by default.
Review Comment:
File committer is still the default committer, right? So what does magic committer being "enabled by default" here mean?
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -530,18 +527,22 @@ performance.
Review Comment:
nit: blank lines
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -180,20 +180,20 @@ and restarting the job.
whose output is in the job attempt directory, *and only rerunning all uncommitted tasks*.
-This algorithm does not works safely or swiftly with AWS S3 storage because
-tenames go from being fast, atomic operations to slow operations which can fail partway through.
+This algorithm does not work safely or swiftly with AWS S3 storage because
+renames go from being fast, atomic operations to slow operations which can fail partway through.
This then is the problem which the S3A committers address:
-*How to safely and reliably commit work to Amazon S3 or compatible object store*
+>*How to safely and reliably commit work to Amazon S3 or compatible object store.*
Review Comment:
What are we quoting here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1161515758
Note that with the current patch, the table of contents has changed.
FROM (trunk):
```markdown
# Committing work to S3 with the "S3A Committers"
### January 2021 Update
## Introduction: The Commit Problem
### Background : Hadoop's "Commit Protocol"
## Meet the S3A Committers
### The Staging Committer
## Conflict Resolution in the Staging Committers
### The Magic Committer
#### Which Committer to Use?
## Switching to an S3A Committer
## Using the Directory and Partitioned Staging Committers
## The "Partitioned" Staging Committer
### Notes
## Using the Magic committer
### FileSystem client setup
### Enabling the committer
## Common Committer Options
## Staging committer (Directory and Partitioned) options
### Common Committer Options
### Staging Committer Options
### Disabling magic committer path rewriting
## <a name="concurrent-jobs"></a> Concurrent Jobs writing to the same destination
## Troubleshooting
```
TO (HEAD):
```markdown
# Committing work to S3 with the "S3A Committers"
### January 2021 Update
## Introduction: The Commit Problem
### Background: Hadoop's "Commit Protocol"
## Meet the S3A Committers
### The Staging Committers
#### Conflict Resolution in the Staging Committers
### The Magic Committer
### Which Committer to Use?
## Switching to an S3A Committer
## Using the Staging Committers
### The "Partitioned" Staging Committer
### Notes on using Staging Committers
## Using the Magic committer
### FileSystem client setup
### Enabling the committer
## Committer Options Reference
### Common S3A Committer Options
### Staging committer (Directory and Partitioned) options
### Disabling magic committer path rewriting
## <a name="concurrent-jobs"></a> Concurrent Jobs writing to the same destination
## Troubleshooting
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
mehakmeet commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1283554642
Thanks @dannycjones, Can you please open a backport to branch-3.3, I'll hit merge after green yetus there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1164304062
@ahmarsuhail, I've updated based on your feedback. Thanks for reviewing it with such detail.
I've put the changes for things like missing `.` in separate commit cd600503542751a72b7b21028ef08b26e41e6580 in case we don't want to touch too many lines unnecessarily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ahmarsuhail commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
ahmarsuhail commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r902757105
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -88,17 +88,17 @@ proportional to the amount of data created. It still can't handle task failure.
loss or corruption of generated data**
-To address these problems there is now explicit support in the `hadop-aws`
-module for committing work to Amazon S3 via the S3A filesystem client,
-*the S3A Committers*
+To address these problems there is now explicit support in the `hadoop-aws`
+module for committing work to Amazon S3 via the S3A filesystem client:
+*the S3A Committers*.
For safe, as well as high-performance output of work to S3,
-we need use "a committer" explicitly written to work with S3, treating it as
-an object store with special features.
+we need to use "a committer" explicitly written to work with S3,
+treating it as an object store with special features.
-### Background : Hadoop's "Commit Protocol"
+### Background: Hadoop's "Commit Protocol"
How exactly is work written to its final destination? That is accomplished by
a "commit protocol" between the workers and the job manager.
Review Comment:
line 112 has a typo. The job has "workers", which are processes which work _with_ the actual data
and write the results.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -88,17 +88,17 @@ proportional to the amount of data created. It still can't handle task failure.
loss or corruption of generated data**
Review Comment:
On line 84, change *. to *
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -165,6 +165,7 @@ that the network has partitioned and that they must abort their work.
That's "essentially" it. When working with HDFS and similar filesystems,
Review Comment:
two full stops on line 146, no full stop on 109, 147.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -283,40 +281,37 @@ new data to an existing partitioned directory tree is a common operation.
</property>
```
-**replace** : when the job is committed (and not before), delete files in
+The _Directory Committer_ uses the entire directory tree for conflict resolution.
+For this committer, the behavior of each conflict mode is shown below:
+
Review Comment:
is there a default mode? if yes can we say which one that is here
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -530,18 +527,22 @@ performance.
### Enabling the committer
+Set the committer used by S3A's committer factory to `magic`:
+
```xml
Review Comment:
unrelated, but do you know why these configs aren't listed in https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#general-s3a-client-configuration? there are other properties (eg: delegation token config) that isn't there either. Wondering if it's useful to have them all in one place so it's easy to see everything that's available
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -696,10 +692,9 @@ The magic committer recognizes when files are created under paths with `__magic/
and redirects the upload to a different location, adding the information needed to complete the upload
Review Comment:
full stop on line 675
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -650,7 +639,14 @@ Conflict management is left to the execution engine itself.
</property>
Review Comment:
full stop/question mark on line 632. little confusing to read currently. also i don't see a config option for ` fs.s3a.committer.staging.uuid` in staging committer options list
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -88,17 +88,17 @@ proportional to the amount of data created. It still can't handle task failure.
loss or corruption of generated data**
-To address these problems there is now explicit support in the `hadop-aws`
-module for committing work to Amazon S3 via the S3A filesystem client,
-*the S3A Committers*
+To address these problems there is now explicit support in the `hadoop-aws`
+module for committing work to Amazon S3 via the S3A filesystem client:
+*the S3A Committers*.
For safe, as well as high-performance output of work to S3,
-we need use "a committer" explicitly written to work with S3, treating it as
-an object store with special features.
+we need to use "a committer" explicitly written to work with S3,
+treating it as an object store with special features.
-### Background : Hadoop's "Commit Protocol"
+### Background: Hadoop's "Commit Protocol"
How exactly is work written to its final destination? That is accomplished by
a "commit protocol" between the workers and the job manager.
Review Comment:
line 109 could possibly be made clearer: A "Job" is the entire query, which takes a given input and produces an output.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -283,40 +281,37 @@ new data to an existing partitioned directory tree is a common operation.
</property>
```
-**replace** : when the job is committed (and not before), delete files in
+The _Directory Committer_ uses the entire directory tree for conflict resolution.
+For this committer, the behavior of each conflict mode is shown below:
+
Review Comment:
looks like it's append based on the config table later, let's mention it here as well.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -68,9 +68,9 @@ process across the cluster may rename a file or directory to the same path.
If the rename fails for any reason, either the data is at the original location,
Review Comment:
missing full stop on line 54.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -180,8 +181,8 @@ and restarting the job.
whose output is in the job attempt directory, *and only rerunning all uncommitted tasks*.
-This algorithm does not works safely or swiftly with AWS S3 storage because
-tenames go from being fast, atomic operations to slow operations which can fail partway through.
+This algorithm does not work safely or swiftly with AWS S3 storage because
+renames go from being fast, atomic operations to slow operations which can fail partway through.
This then is the problem which the S3A committers address:
Review Comment:
consider adding a full stop to 188
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -474,7 +466,7 @@ files which do not contain relevant data.
What the partitioned committer does is, where the tooling permits, allows callers
to add data to an existing partitioned layout*.
Review Comment:
Consider rephrasing. "If tool permits, the partitioned committer allows callers to add data to an existing partitioned layout." also remove * before the full stop
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -492,18 +484,19 @@ was written. With the policy of `append`, the new file would be added to
the existing set of files.
-### Notes
+### Notes on using Staging Committers
1. A deep partition tree can itself be a performance problem in S3 and the s3a client,
-or, more specifically. a problem with applications which use recursive directory tree
+or more specifically a problem with applications which use recursive directory tree
walks to work with data.
1. The outcome if you have more than one job trying simultaneously to write data
to the same destination with any policy other than "append" is undefined.
1. In the `append` operation, there is no check for conflict with file names.
-If, in the example above, the file `log-20170228.avro` already existed,
-it would be overridden. Set `fs.s3a.committer.staging.unique-filenames` to `true`
+If the file `log-20170228.avro` in the example above already existed, it would be overwritten.
+
+ Set `fs.s3a.committer.staging.unique-filenames` to `true`
Review Comment:
Indentation isn't right here. I think it makes sense to have this as part of point 3, consider reverting this change
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -88,17 +88,17 @@ proportional to the amount of data created. It still can't handle task failure.
loss or corruption of generated data**
Review Comment:
add a full stop to line 88
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -88,17 +88,17 @@ proportional to the amount of data created. It still can't handle task failure.
loss or corruption of generated data**
-To address these problems there is now explicit support in the `hadop-aws`
-module for committing work to Amazon S3 via the S3A filesystem client,
-*the S3A Committers*
+To address these problems there is now explicit support in the `hadoop-aws`
+module for committing work to Amazon S3 via the S3A filesystem client:
+*the S3A Committers*.
For safe, as well as high-performance output of work to S3,
-we need use "a committer" explicitly written to work with S3, treating it as
-an object store with special features.
+we need to use "a committer" explicitly written to work with S3,
+treating it as an object store with special features.
-### Background : Hadoop's "Commit Protocol"
+### Background: Hadoop's "Commit Protocol"
How exactly is work written to its final destination? That is accomplished by
a "commit protocol" between the workers and the job manager.
Review Comment:
Typo on line 129. Guessing it should be a separate bullet point
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -165,6 +165,7 @@ that the network has partitioned and that they must abort their work.
That's "essentially" it. When working with HDFS and similar filesystems,
Review Comment:
I think it makes sense to swap lines 147 and 145. As individual workers will communicate with the job manager first, and then it will make a decision to commit or abort
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -530,18 +527,22 @@ performance.
### Enabling the committer
+Set the committer used by S3A's committer factory to `magic`:
+
```xml
Review Comment:
ok they're listed here: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/bk_cloud-data-access/content/s3-config-parameters.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r904911780
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -474,7 +466,7 @@ files which do not contain relevant data.
What the partitioned committer does is, where the tooling permits, allows callers
to add data to an existing partitioned layout*.
Review Comment:
for this one, i'm not sure if the `*` is referring to something. I will remove for now. Happy to add it back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1164488192
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 4m 13s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/4/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| +1 :green_heart: | mvnsite | 4m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 63m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4478 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets markdownlint |
| uname | Linux dbf5984762f5 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cd97bad7950f99e1611887b8e8d4c93e75c8bf7d |
| Max. process+thread count | 608 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/4/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1161744669
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 69m 9s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 19s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. |
| | | 97m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4478 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets markdownlint |
| uname | Linux 8c6d918a1efa 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ffdc6bd7f763343b651d7a5b86a4fa2fa2f260eb |
| Max. process+thread count | 567 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r904894123
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -283,40 +281,37 @@ new data to an existing partitioned directory tree is a common operation.
</property>
```
-**replace** : when the job is committed (and not before), delete files in
+The _Directory Committer_ uses the entire directory tree for conflict resolution.
+For this committer, the behavior of each conflict mode is shown below:
+
Review Comment:
I will add it above the XML example. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r904905036
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -165,6 +165,7 @@ that the network has partitioned and that they must abort their work.
That's "essentially" it. When working with HDFS and similar filesystems,
Review Comment:
I agree - I will actually remove redundant "whether the output is to be *committed* or *aborted*" to hopefully make it clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#issuecomment-1161595695
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 64m 54s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/1/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. |
| | | 92m 26s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4478 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets markdownlint |
| uname | Linux f71388264706 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5e8cdf09694ef5b1a2f1cd33016a068ea9bd469c |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4478/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on a diff in pull request #4478: HADOOP-18304. Improve user-facing S3A committers documentation
Posted by GitBox <gi...@apache.org>.
dannycjones commented on code in PR #4478:
URL: https://github.com/apache/hadoop/pull/4478#discussion_r989873550
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -530,18 +527,22 @@ performance.
Review Comment:
will drop those!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org