You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/06/25 12:00:00 UTC
[jira] [Work logged] (HADOOP-17774) bytesRead FS statistic showing
twice the correct value in S3A
[ https://issues.apache.org/jira/browse/HADOOP-17774?focusedWorklogId=614953&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-614953 ]
ASF GitHub Bot logged work on HADOOP-17774:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 25/Jun/21 11:59
Start Date: 25/Jun/21 11:59
Worklog Time Spent: 10m
Work Description: mehakmeet opened a new pull request #3144:
URL: https://github.com/apache/hadoop/pull/3144
Test command: ```mvn clean verify -Dparallel-tests -DtestsThreadCount=4 -Dscale```
Region: ap-south-1
```
INFO] Results:
[INFO]
[WARNING] Tests run: 568, Failures: 0, Errors: 0, Skipped: 5
```
```
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut
[ERROR] ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut
[INFO]
[ERROR] Tests run: 1460, Failures: 0, Errors: 2, Skipped: 462
```
```
[ERROR] Errors:
[ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:267 » TestTimedOut
[INFO]
[ERROR] Tests run: 151, Failures: 2, Errors: 1, Skipped: 28
```
Seeing these errors:
```
[ERROR] Failures:
[ERROR] ITestS3AContractRootDir.testListEmptyRootDirectory:82->AbstractContractRootDirectoryTest.testListEmptyRootDirectory:196->Assert.fail:89 Deleted file: unexpectedly found s3a://mehakmeet-singh-data/user as S3AFileStatus{path=s3a://mehakmeet-singh-data/user; isDirectory=true; modification_time=0; access_time=0; owner=mehakmeet.singh; group=mehakmeet.singh; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null
[ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:101->Assert.fail:89 After 20 attempts: listing after rm /* not empty
final [00] S3AFileStatus{path=s3a://mehakmeet-singh-data/user; isDirectory=true; modification_time=0; access_time=0; owner=mehakmeet.singh; group=mehakmeet.singh; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null
```
have seen these errors intermittently, due to some issue with DynamoDB table.
@steveloughran suggested ```hadoop org.apache.hadoop.fs.s3a.s3guard.PurgeS3GuardDynamoTable -force s3a://example-bucket/```
But that fails with an error:
```
2021-06-25 16:18:43,464 INFO service.AbstractService: Service PurgeS3GuardDynamoTable failed in state STARTED
-1: Filesystem has no metadata store: s3a://mehakmeet-singh-data
at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic.failure(AbstractS3GuardDynamoDBDiagnostic.java:115)
at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic.require(AbstractS3GuardDynamoDBDiagnostic.java:94)
at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic.bindStore(AbstractS3GuardDynamoDBDiagnostic.java:157)
at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic.bindFromCLI(AbstractS3GuardDynamoDBDiagnostic.java:147)
at org.apache.hadoop.fs.s3a.s3guard.PurgeS3GuardDynamoTable.serviceStart(PurgeS3GuardDynamoTable.java:123)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.service.launcher.ServiceLauncher.coreServiceLaunch(ServiceLauncher.java:619)
at org.apache.hadoop.service.launcher.ServiceLauncher.launchService(ServiceLauncher.java:494)
at org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.serviceMain(DumpS3GuardDynamoTable.java:517)
at org.apache.hadoop.fs.s3a.s3guard.PurgeS3GuardDynamoTable.main(PurgeS3GuardDynamoTable.java:205)
2021-06-25 16:18:43,467 INFO util.ExitUtil: Exiting with status -1: Filesystem has no metadata store: s3a://mehakmeet-singh-data
```
Would like the reviewers to run the aws test suite once in their setup while reviewing as well.
CC: @steveloughran @mukund-thakur @bogthe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 614953)
Remaining Estimate: 0h
Time Spent: 10m
> bytesRead FS statistic showing twice the correct value in S3A
> -------------------------------------------------------------
>
> Key: HADOOP-17774
> URL: https://issues.apache.org/jira/browse/HADOOP-17774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> S3A "bytes read" statistic is being incremented twice. Firstly while reading in S3AInputStream and then in merge() of S3AInstrumentation when S3AInputStream is closed.
> This makes "bytes read" statistic equal to sum of stream_read_bytes and stream_read_total_bytes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org