You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2019/01/18 22:14:00 UTC
[jira] [Commented] (HADOOP-15890) Some S3A committer tests don't
match ITest* pattern; don't run in maven
[ https://issues.apache.org/jira/browse/HADOOP-15890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16746723#comment-16746723 ]
Steve Loughran commented on HADOOP-15890:
-----------------------------------------
{code}
bin/hadoop fs -cat s3a://hwdev-steve-ireland-new/terasort-ITestMagicTerasort/results.csv
"Operation" "Duration"
"Generate" "0:22.854s"
"Terasort" "0:30.228s"
"Validate" "0:27.682s"
"Completed" "1:27.840s"
{code}
Directory staging committer:
{code}
bin/hadoop fs -cat s3a://hwdev-steve-ireland-new/terasort-ITestDirectoryTerasort/results.csv
"Operation" "Duration"
"Generate" "0:22.111s"
"Terasort" "0:24.613s"
"Validate" "0:24.504s"
"Completed" "1:19.135s"
{code}
The client is a laptop, store S3 ireland a few hundred millis away; the latency of S3 calls means that the magic committer, which uses S3 over a miniHDFS cluster, suffers a lot from the latency. You'd need to be creating larger files for the incremental write to become relevant
> Some S3A committer tests don't match ITest* pattern; don't run in maven
> -----------------------------------------------------------------------
>
> Key: HADOOP-15890
> URL: https://issues.apache.org/jira/browse/HADOOP-15890
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.2.0, 3.1.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> some of the s3A committer tests don't have the right prefix for the maven IT test runs to pick up
> {code}
> ITMagicCommitMRJob.java
> ITStagingCommitMRJobBad
> ITDirectoryCommitMRJob
> ITStagingCommitMRJob
> {code}
> They all work when run by name or in the IDE (which is where I developed them), but they don't run in maven builds.
> Fix: rename. There are some new tests in branch-3.2 from HADOOP-15107 which aren't in 3.1; need patches for both.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org