You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Zoltán Borók-Nagy (Jira)" <ji...@apache.org> on 2020/03/04 15:07:00 UTC
[jira] [Updated] (IMPALA-9461) S3Guard metadata store cannot be
found
[ https://issues.apache.org/jira/browse/IMPALA-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zoltán Borók-Nagy updated IMPALA-9461:
--------------------------------------
Description:
I've seen this in a build on S3 + S3Guard. Many tests failed because S3Guard metadata store didn't exist:
{noformat}
I0304 00:34:48.754648 17557 status.cc:126] couldn't retrieve FileSystem:
DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off{noformat}
In impalad.ERROR:
{noformat}
hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
ResourceNotFoundException: Requested resource not found: Table: impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 6A5PINL6IDFF93BNDSSOQCG6JFVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io. FileNotFoundException: DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStoreTableManager.initTable(DynamoDBMetadataStoreTableManager.java:216)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:427)
at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:109)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:418)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3320)
{noformat}
I suspected that the S3Guard metadata store initialization failed, so I checked load-test-warehouse-snapshot.log, but seems like S3Guard init was successful:
{noformat}
20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb
20/03/03 22:03:38 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized.
Metadata Store Diagnostics:
ARN=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1
billing-mode=per-request
description=S3Guard metadata store in DynamoDB
name=impala-test-uswest2-1
persist.authoritative.bit=true
read-capacity=0
region=us-west-2
retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS)
size=9791493
status=ACTIVE
table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: impala-test-uswest2-1,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE, CreationDateTime: Mon Mar 02 22:02:44 PST 2020,ProvisionedThroughput: {NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits: 0},TableSizeBytes: 9791493,ItemCount: 47304,TableArn: arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1,TableId: c0fad823-9fb7-48ba-b803-661a2b4052ee,BillingModeSummary: {BillingMode: PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 02 22:02:44 PST 2020},}
write-capacity=0
20/03/03 22:03:38 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb
20/03/03 22:03:40 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized.
20/03/03 22:03:40 INFO s3guard.DynamoDBMetadataStore: Starting: Pruning DynamoDB Store
20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Pruning DynamoDB Store: duration 1:38.538s
20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Finished pruning 25981 items in batches of 25
{noformat}
So based on this impala-test-uswest2-1 is successfully created in the region us-west-2, but later it became unreachable/deleted.
was:
I've seen this in a build on S3 + S3Guard. Many tests failed because S3Guard metadata store didn't exist:
{noformat}
I0304 00:34:48.754648 17557 status.cc:126] couldn't retrieve FileSystem:
DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off{noformat}
In impalad.ERROR:
{noformat}
ResourceNotFoundException: Requested resource not found: Table: impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: C9PEDOBGV9AIT482FV3RMDEA4NVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io.FileNotFoundException: DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned offResourceNotFoundException: Requested resource not found: Table: impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: C9PEDOBGV9AIT482FV3RMDEA4NVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io.FileNotFoundException: DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStoreTableManager.initTable(DynamoDBMetadataStoreTableManager.java:216) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:427) at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:109) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:418){noformat}
I suspected that the S3Guard metadata store initialization failed, so I checked load-test-warehouse-snapshot.log, but seems like S3Guard init was successful:
{noformat}
20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb20/03/03 22:03:38 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized.Metadata Store Diagnostics: ARN=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1 billing-mode=per-request description=S3Guard metadata store in DynamoDB name=impala-test-uswest2-1 persist.authoritative.bit=true read-capacity=0 region=us-west-2 retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS) size=9791493 status=ACTIVE table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: impala-test-uswest2-1,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Mon Mar 02 22:02:44 PST 2020,ProvisionedThroughput: {NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits: 0},TableSizeBytes: 9791493,ItemCount: 47304,TableArn: arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1,TableId: c0fad823-9fb7-48ba-b803-661a2b4052ee,BillingModeSummary: {BillingMode: PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 02 22:02:44 PST 2020},} write-capacity=020/03/03 22:03:38 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb20/03/03 22:03:40 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized.20/03/03 22:03:40 INFO s3guard.DynamoDBMetadataStore: Starting: Pruning DynamoDB Store20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Pruning DynamoDB Store: duration 1:38.538s20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Finished pruning 25981 items in batches of 25{noformat}
So based on this impala-test-uswest2-1 is successfully created in the region us-west-2, but later it became unreachable/deleted.
> S3Guard metadata store cannot be found
> --------------------------------------
>
> Key: IMPALA-9461
> URL: https://issues.apache.org/jira/browse/IMPALA-9461
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Priority: Major
> Labels: broken-build
>
> I've seen this in a build on S3 + S3Guard. Many tests failed because S3Guard metadata store didn't exist:
> {noformat}
> I0304 00:34:48.754648 17557 status.cc:126] couldn't retrieve FileSystem:
> DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off{noformat}
> In impalad.ERROR:
> {noformat}
> hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
> ResourceNotFoundException: Requested resource not found: Table: impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 6A5PINL6IDFF93BNDSSOQCG6JFVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io. FileNotFoundException: DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off
> at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStoreTableManager.initTable(DynamoDBMetadataStoreTableManager.java:216)
> at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:427)
> at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:109)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:418)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3320)
> {noformat}
> I suspected that the S3Guard metadata store initialization failed, so I checked load-test-warehouse-snapshot.log, but seems like S3Guard init was successful:
> {noformat}
> 20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb
> 20/03/03 22:03:38 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized.
> Metadata Store Diagnostics:
> ARN=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1
> billing-mode=per-request
> description=S3Guard metadata store in DynamoDB
> name=impala-test-uswest2-1
> persist.authoritative.bit=true
> read-capacity=0
> region=us-west-2
> retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS)
> size=9791493
> status=ACTIVE
> table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: impala-test-uswest2-1,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE, CreationDateTime: Mon Mar 02 22:02:44 PST 2020,ProvisionedThroughput: {NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits: 0},TableSizeBytes: 9791493,ItemCount: 47304,TableArn: arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1,TableId: c0fad823-9fb7-48ba-b803-661a2b4052ee,BillingModeSummary: {BillingMode: PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 02 22:02:44 PST 2020},}
> write-capacity=0
> 20/03/03 22:03:38 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb
> 20/03/03 22:03:40 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized.
> 20/03/03 22:03:40 INFO s3guard.DynamoDBMetadataStore: Starting: Pruning DynamoDB Store
> 20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Pruning DynamoDB Store: duration 1:38.538s
> 20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Finished pruning 25981 items in batches of 25
> {noformat}
> So based on this impala-test-uswest2-1 is successfully created in the region us-west-2, but later it became unreachable/deleted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org