You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "CandiceSu (via GitHub)" <gi...@apache.org> on 2024/12/20 18:19:39 UTC
[I] AvroParquetWriter cache old aws config even after close, clean and create new writer [parquet-java]
CandiceSu opened a new issue, #3106:
URL: https://github.com/apache/parquet-java/issues/3106
### Describe the bug, including details regarding any error messages, version, and platform.
Hi we created one AvroParquetWriter to write to aws s3.
This is how we create the writer:
`ParquetWriter<GenericRecord> writer = AvroParquetWriter.<GenericRecord>builder(path)
.withConf(conf)
.withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
.withSchema(avroSchema)
.build()`
And this is how we create the conf:
` AwsCredentialsProvider credentialsProvider = DefaultCredentialsProvider.builder().build();
var credentials = credentialsProvider.resolveCredentials();
AwsSessionCredentials sessionCredentials = (AwsSessionCredentials) credentials;
Configuration conf = new Configuration();
conf.set("fs.s3a.access.key", sessionCredentials.accessKeyId());
conf.set("fs.s3a.secret.key", sessionCredentials.secretAccessKey());
conf.set("fs.s3a.session.token", sessionCredentials.sessionToken());
conf.set("fs.s3a.endpoint", "s3.my_region.amazonaws.com");
`
And we are running a streaming application in ECS, each time a task come in, we get a new conf and create a new writer.
We see the token and everything are correct and refreshed each time, but writer fail to write to s3 and throw 'software.amazon.awssdk.services.s3.model.S3Exception: The provided token has expired' after the application running for few hours.
So it looks like the writer, once initialized 1st time, it keep using the 1st time configuration - even if we closed the writer, cleared cache, create new writer with new conf... And after few hours of course you hit token expire issue when that 1st time token expires.
We even did a small test - create writer 1st time with correct conf, write to s3 works. Close 1st writer, create writer 2nd time with invalid conf, write to s3 still works!!!
Is there anything wrong in the way we initialize the writer or conf?
Thanks!
### Component(s)
Avro
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] AvroParquetWriter cache old aws config even after close, clean and create new writer [parquet-java]
Posted by "CandiceSu (via GitHub)" <gi...@apache.org>.
CandiceSu commented on issue #3106:
URL: https://github.com/apache/parquet-java/issues/3106#issuecomment-2563932468
thx @wgtmac
we find out the issue
it's not the avro parquet writer itself caching the token, its the conf object keep using same token after initialized the 1st time.
Configuration should set "fs.s3a.aws.credentials.provider", otherwise it uses default TemporaryAWSCredentialsProvider which extends AbstractSessionCredentialsProvider which keeps using the same credentials once initialized the 1st time:
`
private final AtomicBoolean initialized = new AtomicBoolean(false);
`
set provider fixed the issue:
`
conf.set("fs.s3a.aws.credentials.provider","software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider");
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] AvroParquetWriter cache old aws config even after close, clean and create new writer [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #3106:
URL: https://github.com/apache/parquet-java/issues/3106#issuecomment-2561596162
I don't think the Parquet writer caches it. Could you please try to see if `fs.s3.impl.disable.cache=false` fixes your issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] AvroParquetWriter cache old aws config even after close, clean and create new writer [parquet-java]
Posted by "CandiceSu (via GitHub)" <gi...@apache.org>.
CandiceSu closed issue #3106: AvroParquetWriter cache old aws config even after close, clean and create new writer
URL: https://github.com/apache/parquet-java/issues/3106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org