You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2018/04/27 15:41:00 UTC
[jira] [Commented] (HADOOP-15422) s3guard doesn't list root dir
when the secrets are in the s3a URI
[ https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456625#comment-16456625 ]
Steve Loughran commented on HADOOP-15422:
-----------------------------------------
The first stack was a spurious failure, but with the corrected credentials, an ls of the root directory fails saying the child path must be a child of the path-with-secrets-inline
{code}
Listing table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
-ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of s3a://A..A:X..X@hwdev-steve-2/
{code}
. Listing of child paths which exist/don't exist pass
Other commands (get, diff) work and don't include secrets in their listing
This is probably related to our work to strip out secrets from URIs. We don't want them in the database, or in logs
{code}
> hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/"
...
{code}
18/04/27 15:21:49 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client in region us-west-1
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Binding to table hwdev-steve-2
18/04/27 15:21:49 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 59156,ItemCount: 432,TableArn: arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 11eb665e-746c-4255-8cc1-6890b0bbea24,}
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Using metadata store DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, authoritative=false
18/04/27 15:21:49 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 86400
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1 -> 1
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 1
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/ ()
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: List status for path: s3a://A..A:X..X@hwdev-steve-2/
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_list_status += 1 -> 1
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 2
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/ ()
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Listing table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
-ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of s3a://A..A:X..X@hwdev-steve-2/
..
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]
[-cat [-ignoreCrc] <src> ...]
[-checksum <src> ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst>]
[-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
? echo $?
255
{code}
Works for a child entry which exists
{code}
[root@ctr-e138-1518143905142-264443-01-000006 ~]# hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/cloud-integration"
18/04/27 15:24:29 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for hwdev-steve-2
18/04/27 15:24:29 DEBUG s3a.S3AUtils: Propagating entries under fs.s3a.bucket.hwdev-steve-2.
18/04/27 15:24:29 WARN s3native.S3xLoginHelper: The Filesystem URI contains login details. This is insecure and may be unsupported in future.
18/04/27 15:24:29 DEBUG s3a.S3AUtils: For URI s3a://hwdev-steve-2//cloud-integration, using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider com.amazonaws.auth.InstanceProfileCredentialsProvider@14008db3
...
18/04/27 15:24:30 DEBUG s3guard.S3Guard: Using DynamoDBMetadataStore metadata store for s3a filesystem
18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Inferring DynamoDB region from S3 bucket: us-west-1
18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Creating DynamoDB client class org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory$DefaultDynamoDBClientFactory with S3 region us-west-1
18/04/27 15:24:30 DEBUG s3a.S3AUtils: For URI (null URI), using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider com.amazonaws.auth.InstanceProfileCredentialsProvider@14008db3
18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.maximum is 15
18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.attempts.maximum is 20
18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.establish.timeout is 5000
18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.timeout is 200000
18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.socket.send.buffer is 8192
18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.socket.recv.buffer is 8192
18/04/27 15:24:30 DEBUG s3a.S3AFileSystem: Using User-Agent: User-Agent: APN/1.0 Hortonworks/1.0 HDP/3.0.0.0-1252, Hadoop 3.0.0.3.0.0.0-1252
18/04/27 15:24:30 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client in region us-west-1
18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Binding to table hwdev-steve-2
18/04/27 15:24:30 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider
18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 59156,ItemCount: 432,TableArn: arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 11eb665e-746c-4255-8cc1-6890b0bbea24,}
18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017
18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Using metadata store DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, authoritative=false
18/04/27 15:24:31 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 86400
18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1 -> 1
18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 1
18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/cloud-integration (cloud-integration)
18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/cloud-integration
18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1 returning for s3a://A..A:X..X@hwdev-steve-2/cloud-integration: PathMetadata{fileStatus=FileStatus{path=s3a://hwdev-steve-2/cloud-integration; isDirectory=true; modification_time=1524842671173; access_time=0; owner=root; group=root; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}; isEmptyDirectory=UNKNOWN; isDeleted=false}
18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: List status for path: s3a://hwdev-steve-2/cloud-integration
18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_list_status += 1 -> 1
18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 2
18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Getting path status for s3a://hwdev-steve-2/cloud-integration (cloud-integration)
18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://hwdev-steve-2/cloud-integration
18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1 returning for s3a://hwdev-steve-2/cloud-integration: PathMetadata{fileStatus=FileStatus{path=s3a://hwdev-steve-2/cloud-integration; isDirectory=true; modification_time=1524842671185; access_time=0; owner=root; group=root; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}; isEmptyDirectory=UNKNOWN; isDeleted=false}
18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Listing table hwdev-steve-2 in region us-west-1: s3a://hwdev-steve-2/cloud-integration
18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory cloud-integration/
18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: object_list_requests += 1 -> 1
18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Adding directory: S3AFileStatus{path=s3a://hwdev-steve-2/cloud-integration/DELAY_LISTING_ME; isDirectory=true; modification_time=0; access_time=0; owner=root; group=root; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false} isEmptyDirectory=FALSE
18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Added 1 entries; ignored 0; hasNext=true; hasMoreObjects=false
18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Start iterating the provided status.
Found 1 items
drwxrwxrwx - root root 0 2018-04-27 15:24 s3a://hwdev-steve-2/cloud-integration/DELAY_LISTING_ME
18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Filesystem s3a://hwdev-steve-2 is closed
18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Shutting down DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}
{code}
And for one which doesn;t
{code}
# hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/missing"
18/04/27 15:25:21 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for hwdev-steve-2
...
18/04/27 15:25:23 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client in region us-west-1
18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Binding to table hwdev-steve-2
18/04/27 15:25:23 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider
18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 59156,ItemCount: 432,TableArn: arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 11eb665e-746c-4255-8cc1-6890b0bbea24,}
18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017
18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Using metadata store DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, authoritative=false
18/04/27 15:25:23 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 86400
18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1 -> 1
18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 1
18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/missing (missing)
18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/missing
18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1 returning for s3a://A..A:X..X@hwdev-steve-2/missing: null
18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1 -> 1
18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1 -> 2
18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: object_list_requests += 1 -> 1
18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Not Found: s3a://A..A:X..X@hwdev-steve-2/missing
ls: `s3a://A..A:X..X@hwdev-steve-2/missing': No such file or directory
18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Filesystem s3a://hwdev-steve-2 is closed
18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Shutting down DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}
{code}
> s3guard doesn't list root dir when the secrets are in the s3a URI
> -----------------------------------------------------------------
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Priority: Minor
>
> If the AWS secrets are in the login, S3guard doesn't list the root dir
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org