You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by zshamrock <al...@gmail.com> on 2017/01/02 15:59:12 UTC
Old AWS SDK version, why?
Why does Apache Ignite still depend on the old AWS SDK version
/<aws.sdk.version>1.10.29</aws.sdk.version>/.
Are there any technical reasons to do so? What if I explicitly exclude those
dependencies, and use the latest 1.11.76, would it work?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by chandresh pancholi <ch...@gmail.com>.
I have tested on AWS with my account but for GCE i will test it on TeamCity.
On Wed, Jan 4, 2017 at 11:36 PM, Denis Magda <dm...@apache.org> wrote:
> Chandresh, thanks for taking this over!
>
> Please update all the modules then which are ignite-aws, ignite-gce and
> ignite-clouds. The modules are listed in the ticket.
>
> As for testing, you need to run corresponding tests on TeamCity which has
> all required keys. Otherwise, you need to create a personal account on AWS
> and GCE if you want to validate the changes locally.
>
> —
> Denis
>
> > On Jan 4, 2017, at 12:50 AM, chandresh pancholi <
> chandreshpancholi007@gmail.com> wrote:
> >
> > I am working on it. I have started. Its asking for aws key.
> >
> > On Tue, Jan 3, 2017 at 11:04 PM, Denis Magda <dm...@apache.org> wrote:
> >
> >> CC-ing dev@ list.
> >>
> >> Created the ticket to update all 3rd party dependencies for our cloud
> >> based IP finders (AWS, GCE and JClouds).
> >> https://issues.apache.org/jira/browse/IGNITE-4519 <
> >> https://issues.apache.org/jira/browse/IGNITE-4519>
> >>
> >> Is there anyone in community who are interested in handling this?
> >>
> >> —
> >> Denis
> >>
> >>> On Jan 2, 2017, at 12:11 PM, vkulichenko <
> valentin.kulichenko@gmail.com>
> >> wrote:
> >>>
> >>> This is just a version ignite-aws module was developed on, and I don't
> >> think
> >>> newer versions were ever tested. However, I believe it should work with
> >>> 1.11.76. Can you try it out and let us know if it does?
> >>>
> >>> -Val
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context: http://apache-ignite-users.
> >> 70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9825.html
> >>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > Chandresh Pancholi
> > Senior Software Engineer
> > Flipkart.com
> > Email-id:chandresh.pancholi@flipkart.com
> > Contact:08951803660
>
>
--
Chandresh Pancholi
Senior Software Engineer
Flipkart.com
Email-id:chandresh.pancholi@flipkart.com
Contact:08951803660
Re: Old AWS SDK version, why?
Posted by Denis Magda <dm...@apache.org>.
Chandresh, thanks for taking this over!
Please update all the modules then which are ignite-aws, ignite-gce and ignite-clouds. The modules are listed in the ticket.
As for testing, you need to run corresponding tests on TeamCity which has all required keys. Otherwise, you need to create a personal account on AWS and GCE if you want to validate the changes locally.
—
Denis
> On Jan 4, 2017, at 12:50 AM, chandresh pancholi <ch...@gmail.com> wrote:
>
> I am working on it. I have started. Its asking for aws key.
>
> On Tue, Jan 3, 2017 at 11:04 PM, Denis Magda <dm...@apache.org> wrote:
>
>> CC-ing dev@ list.
>>
>> Created the ticket to update all 3rd party dependencies for our cloud
>> based IP finders (AWS, GCE and JClouds).
>> https://issues.apache.org/jira/browse/IGNITE-4519 <
>> https://issues.apache.org/jira/browse/IGNITE-4519>
>>
>> Is there anyone in community who are interested in handling this?
>>
>> —
>> Denis
>>
>>> On Jan 2, 2017, at 12:11 PM, vkulichenko <va...@gmail.com>
>> wrote:
>>>
>>> This is just a version ignite-aws module was developed on, and I don't
>> think
>>> newer versions were ever tested. However, I believe it should work with
>>> 1.11.76. Can you try it out and let us know if it does?
>>>
>>> -Val
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-ignite-users.
>> 70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9825.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Chandresh Pancholi
> Senior Software Engineer
> Flipkart.com
> Email-id:chandresh.pancholi@flipkart.com
> Contact:08951803660
Re: Old AWS SDK version, why?
Posted by Denis Magda <dm...@apache.org>.
Chandresh, thanks for taking this over!
Please update all the modules then which are ignite-aws, ignite-gce and ignite-clouds. The modules are listed in the ticket.
As for testing, you need to run corresponding tests on TeamCity which has all required keys. Otherwise, you need to create a personal account on AWS and GCE if you want to validate the changes locally.
—
Denis
> On Jan 4, 2017, at 12:50 AM, chandresh pancholi <ch...@gmail.com> wrote:
>
> I am working on it. I have started. Its asking for aws key.
>
> On Tue, Jan 3, 2017 at 11:04 PM, Denis Magda <dm...@apache.org> wrote:
>
>> CC-ing dev@ list.
>>
>> Created the ticket to update all 3rd party dependencies for our cloud
>> based IP finders (AWS, GCE and JClouds).
>> https://issues.apache.org/jira/browse/IGNITE-4519 <
>> https://issues.apache.org/jira/browse/IGNITE-4519>
>>
>> Is there anyone in community who are interested in handling this?
>>
>> —
>> Denis
>>
>>> On Jan 2, 2017, at 12:11 PM, vkulichenko <va...@gmail.com>
>> wrote:
>>>
>>> This is just a version ignite-aws module was developed on, and I don't
>> think
>>> newer versions were ever tested. However, I believe it should work with
>>> 1.11.76. Can you try it out and let us know if it does?
>>>
>>> -Val
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-ignite-users.
>> 70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9825.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Chandresh Pancholi
> Senior Software Engineer
> Flipkart.com
> Email-id:chandresh.pancholi@flipkart.com
> Contact:08951803660
Re: Old AWS SDK version, why?
Posted by chandresh pancholi <ch...@gmail.com>.
I am working on it. I have started. Its asking for aws key.
On Tue, Jan 3, 2017 at 11:04 PM, Denis Magda <dm...@apache.org> wrote:
> CC-ing dev@ list.
>
> Created the ticket to update all 3rd party dependencies for our cloud
> based IP finders (AWS, GCE and JClouds).
> https://issues.apache.org/jira/browse/IGNITE-4519 <
> https://issues.apache.org/jira/browse/IGNITE-4519>
>
> Is there anyone in community who are interested in handling this?
>
> —
> Denis
>
> > On Jan 2, 2017, at 12:11 PM, vkulichenko <va...@gmail.com>
> wrote:
> >
> > This is just a version ignite-aws module was developed on, and I don't
> think
> > newer versions were ever tested. However, I believe it should work with
> > 1.11.76. Can you try it out and let us know if it does?
> >
> > -Val
> >
> >
> >
> > --
> > View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9825.html
> > Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
--
Chandresh Pancholi
Senior Software Engineer
Flipkart.com
Email-id:chandresh.pancholi@flipkart.com
Contact:08951803660
Re: Do We Have Plan To Support Golang Natively?
Posted by Sergi Vladykin <se...@gmail.com>.
Hi,
We actually have C++ bindings for Ignite API, so it must be possible to use
it from Go with cgo and SWIG. Though it may require some additional manual
work and testing.
I welcome you to join Apache Ignite community, contribute this integration
and become its maintainer! :)
Sergi
2017-01-08 8:37 GMT+03:00 刘德忠 <gr...@lottoqq.com>:
> Go is a great language and our future computer language. It is very best
> backend language to cloud distributed system. And ignite is the best
> in-memory database. I think it's time to marry them. Any plan for this?
> Thanks
Do We Have Plan To Support Golang Natively?
Posted by 刘德忠 <gr...@lottoqq.com>.
Go is a great language and our future computer language. It is very best backend language to cloud distributed system. And ignite is the best in-memory database. I think it's time to marry them. Any plan for this? Thanks
Re: Old AWS SDK version, why?
Posted by Denis Magda <dm...@apache.org>.
Thanks Aliaksandr! Granted you the permissions in JIRA.
—
Denis
> On Jan 7, 2017, at 12:15 AM, zshamrock <al...@gmail.com> wrote:
>
> Then, yes, could you, please, assign this ticket to me, or give the necessary
> permissions so I can do it myself, and put it in progress.
>
> Thank you.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9955.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by zshamrock <al...@gmail.com>.
Then, yes, could you, please, assign this ticket to me, or give the necessary
permissions so I can do it myself, and put it in progress.
Thank you.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9955.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by Denis Magda <dm...@apache.org>.
Presently the community plans to release 2.0 in Feb. If it’s shifted there is a chance that 1.9 will be released instead and earlier.
However, someone still needs to take care about this ticket.
—
Denis
> On Jan 6, 2017, at 1:36 PM, zshamrock <al...@gmail.com> wrote:
>
> Hi, Denis.
> Thank you for creating the issue. I will keep an eye on it. If nobody picks
> it up, and I find free time, then I will pick up it myself. When do you
> plan to release 1.9.0 version (is there a roadmap)? (as I've not seen you've
> been releasing minor versions like 1.8.1 including some of the bug fixes or
> you do?).
>
> Regards the credentials provider refreshing the credentials. I would say it
> depends on the implementation, but I've checked a few main used by
> DefaultAWSCredentialsProviderChain:
> public class DefaultAWSCredentialsProviderChain extends
> AWSCredentialsProviderChain {
> public DefaultAWSCredentialsProviderChain() {
> super(new EnvironmentVariableCredentialsProvider(),
> new SystemPropertiesCredentialsProvider(),
> new ProfileCredentialsProvider(),
> new InstanceProfileCredentialsProvider());
> }
> }
>
> all of them always do some extra work in the `getCredentials` method rather
> than just simply return credentials.
>
> Quite interesting implementation ProfileCredentialsProvider has:
> public AWSCredentials getCredentials() {
> if (profilesConfigFile == null) {
> synchronized (this) {
> if (profilesConfigFile == null) {
> profilesConfigFile = new ProfilesConfigFile();
> lastRefreshed = System.nanoTime();
> }
> }
> }
>
> // Periodically check if the file on disk has been modified
> // since we last read it.
> //
> // For active applications, only have one thread block.
> // For applications that use this method in bursts, ensure the
> // credentials are never too stale.
> long now = System.nanoTime();
> long age = now - lastRefreshed;
> if (age > refreshForceIntervalNanos) {
> refresh();
> } else if (age > refreshIntervalNanos) {
> if (refreshSemaphore.tryAcquire()) {
> try {
> refresh();
> } finally {
> refreshSemaphore.release();
> }
> }
> }
>
> return profilesConfigFile.getCredentials(profileName);
> }
>
> where it periodically refreshed the profile file to check if the credentials
> were updated in the meanwhile.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9947.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by zshamrock <al...@gmail.com>.
Also specifically for the Instance Profile credentials provider I guess, on
every getCredentials request, if the credentials are about to expire it
calls instance medata
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials,
i.e.
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<iam
role name>
to obtain new temporal credentials, and so on.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9948.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by zshamrock <al...@gmail.com>.
Hi, Denis.
Thank you for creating the issue. I will keep an eye on it. If nobody picks
it up, and I find free time, then I will pick up it myself. When do you
plan to release 1.9.0 version (is there a roadmap)? (as I've not seen you've
been releasing minor versions like 1.8.1 including some of the bug fixes or
you do?).
Regards the credentials provider refreshing the credentials. I would say it
depends on the implementation, but I've checked a few main used by
DefaultAWSCredentialsProviderChain:
public class DefaultAWSCredentialsProviderChain extends
AWSCredentialsProviderChain {
public DefaultAWSCredentialsProviderChain() {
super(new EnvironmentVariableCredentialsProvider(),
new SystemPropertiesCredentialsProvider(),
new ProfileCredentialsProvider(),
new InstanceProfileCredentialsProvider());
}
}
all of them always do some extra work in the `getCredentials` method rather
than just simply return credentials.
Quite interesting implementation ProfileCredentialsProvider has:
public AWSCredentials getCredentials() {
if (profilesConfigFile == null) {
synchronized (this) {
if (profilesConfigFile == null) {
profilesConfigFile = new ProfilesConfigFile();
lastRefreshed = System.nanoTime();
}
}
}
// Periodically check if the file on disk has been modified
// since we last read it.
//
// For active applications, only have one thread block.
// For applications that use this method in bursts, ensure the
// credentials are never too stale.
long now = System.nanoTime();
long age = now - lastRefreshed;
if (age > refreshForceIntervalNanos) {
refresh();
} else if (age > refreshIntervalNanos) {
if (refreshSemaphore.tryAcquire()) {
try {
refresh();
} finally {
refreshSemaphore.release();
}
}
}
return profilesConfigFile.getCredentials(profileName);
}
where it periodically refreshed the profile file to check if the credentials
were updated in the meanwhile.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9947.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by Denis Magda <dm...@apache.org>.
Thanks for troubleshooting this. I’m absolutely fine to add a method that will accept AWSCredentialsProvider.
Do you want to handle this and improve the IP finder implementation? We can grant you all the needed permissions in JIRA.
https://issues.apache.org/jira/browse/IGNITE-4530 <https://issues.apache.org/jira/browse/IGNITE-4530>
In any case, what I’ve learnt from your explanation is that a provider can refresh credentials. Does this mean that in general you can refresh your credentials manually by creating new/updated ones using AWS interface?
—
Denis
> On Jan 6, 2017, at 5:00 AM, zshamrock <al...@gmail.com> wrote:
>
> Ok, Denis. I found the issue. Could you, please, create the corresponding
> JIRA issue for Apache Ignite project (it looks I don't have enough rights to
> do it myself).
>
> The issue is the following (at least for Apache Ignite 1.7.0, but I guess
> 1.8.0 will not be any different):
>
> - when you deploy Ignite on AWS (in clustered) you need to use
> /TcpDiscoveryS3IpFinder/
> - /TcpDiscoveryS3IpFinder/ only has /setAwsCredentials/ which accepts
> /AWSCredentials/
> - If you check the source code this is how then credentials is used
> /s3 = cfg != null ? new AmazonS3Client(cred, cfg) : new
> AmazonS3Client(cred);/
> inside /TcpDiscoveryS3IpFinder/
> - /AmazonS3Client/ has also another way to construct the client by passing
> /AWSCredentialsProvider/
> /
> public AmazonS3Client(AWSCredentialsProvider credentialsProvider) {
> this(credentialsProvider, new ClientConfiguration());
> }
> /
> - If you pass /AWSCredentials/ in the S3 client instead it wraps in into
> /StaticCredentialsProvider/ instead, i.e.
> /public AmazonS3Client(AWSCredentials awsCredentials, ClientConfiguration
> clientConfiguration) {
> super(clientConfiguration);
> this.awsCredentialsProvider = new
> StaticCredentialsProvider(awsCredentials);
> init();
> }/
> - S3 Amazon client (as all other AWS clients), never use credentials
> directly, but instead call /awsCredentialsProvider.getCredentials()/ every
> time credentials are required
>
> One of the available /AWSCredentialsProvider/ is
> /InstanceProfileCredentialsProvider/ which /getCredentials()/ implementation
> refresh the AWS credentials if required (following EC2 instance profile
> workflow):
> /public AWSCredentials getCredentials() {
> if (needsToLoadCredentials())
> loadCredentials();
> if (expired()) {
> throw new AmazonClientException(
> "The credentials received from the Amazon EC2 metadata
> service have expired");
> }
> return credentials;
> }/
>
> So, the suggestion is to add support for /TcpDiscoveryS3IpFinder/ to accept
> /AWSCredentialsProvider/ in addition to just only /AWSCredentials/, and then
> passing it into underlying /AmazonS3Client/ (actually you can even wrap
> /AWSCredentials/ into /StaticCredentialsProvider/ and always pass
> /AWSCredentialsProvider/ into S3 client).
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9932.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by zshamrock <al...@gmail.com>.
Ok, Denis. I found the issue. Could you, please, create the corresponding
JIRA issue for Apache Ignite project (it looks I don't have enough rights to
do it myself).
The issue is the following (at least for Apache Ignite 1.7.0, but I guess
1.8.0 will not be any different):
- when you deploy Ignite on AWS (in clustered) you need to use
/TcpDiscoveryS3IpFinder/
- /TcpDiscoveryS3IpFinder/ only has /setAwsCredentials/ which accepts
/AWSCredentials/
- If you check the source code this is how then credentials is used
/s3 = cfg != null ? new AmazonS3Client(cred, cfg) : new
AmazonS3Client(cred);/
inside /TcpDiscoveryS3IpFinder/
- /AmazonS3Client/ has also another way to construct the client by passing
/AWSCredentialsProvider/
/
public AmazonS3Client(AWSCredentialsProvider credentialsProvider) {
this(credentialsProvider, new ClientConfiguration());
}
/
- If you pass /AWSCredentials/ in the S3 client instead it wraps in into
/StaticCredentialsProvider/ instead, i.e.
/public AmazonS3Client(AWSCredentials awsCredentials, ClientConfiguration
clientConfiguration) {
super(clientConfiguration);
this.awsCredentialsProvider = new
StaticCredentialsProvider(awsCredentials);
init();
}/
- S3 Amazon client (as all other AWS clients), never use credentials
directly, but instead call /awsCredentialsProvider.getCredentials()/ every
time credentials are required
One of the available /AWSCredentialsProvider/ is
/InstanceProfileCredentialsProvider/ which /getCredentials()/ implementation
refresh the AWS credentials if required (following EC2 instance profile
workflow):
/public AWSCredentials getCredentials() {
if (needsToLoadCredentials())
loadCredentials();
if (expired()) {
throw new AmazonClientException(
"The credentials received from the Amazon EC2 metadata
service have expired");
}
return credentials;
}/
So, the suggestion is to add support for /TcpDiscoveryS3IpFinder/ to accept
/AWSCredentialsProvider/ in addition to just only /AWSCredentials/, and then
passing it into underlying /AmazonS3Client/ (actually you can even wrap
/AWSCredentials/ into /StaticCredentialsProvider/ and always pass
/AWSCredentialsProvider/ into S3 client).
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9932.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by Denis Magda <dm...@apache.org>.
Chandresh has just updated the SDK version and I’ve merged the changes to Ignite’s master branch
https://issues.apache.org/jira/browse/IGNITE-4519 <https://issues.apache.org/jira/browse/IGNITE-4519>
You can build the project from sources and check if the SDK version was a cause of your issue.
—
Denis
> On Jan 4, 2017, at 11:14 AM, zshamrock <al...@gmail.com> wrote:
>
> Thank you for pointing to the Github discussion, Denis.
>
> If I understood it correctly, it doesn't apply to my case (at least
> according to my understanding of the topic). I've checked the expire
> properties of the S3 objects, and they are not set:
>
> Expiry Date: None
> Expiration Rule: N/A
>
> Also I run another services using the same IAM role, and I don't see similar
> error happening for any of the other services in use.
>
> Again, it still can be due to some IAM policy misconfiguration, but I would
> like to try with the latest AWS SDK to be sure (as I expect S3 SDK should
> handle instance profile token expiration by itself, which potentially it
> doesn't).
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9879.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by zshamrock <al...@gmail.com>.
Thank you for pointing to the Github discussion, Denis.
If I understood it correctly, it doesn't apply to my case (at least
according to my understanding of the topic). I've checked the expire
properties of the S3 objects, and they are not set:
Expiry Date: None
Expiration Rule: N/A
Also I run another services using the same IAM role, and I don't see similar
error happening for any of the other services in use.
Again, it still can be due to some IAM policy misconfiguration, but I would
like to try with the latest AWS SDK to be sure (as I expect S3 SDK should
handle instance profile token expiration by itself, which potentially it
doesn't).
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9879.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by Denis Magda <dm...@apache.org>.
Most likely the issue is not related to SDK version. It’s related to your AWS account. Here is a discussion around a similar issue
https://github.com/aws/aws-sdk-ruby/issues/808 <https://github.com/aws/aws-sdk-ruby/issues/808>
In any case, you can quickly check your assumptions in regards to an outdated SDK by changing SDK version in ignite-aws pom.xml directly or by relating the old jars with the new one.
—
Denis
> On Jan 3, 2017, at 11:55 AM, zshamrock <al...@gmail.com> wrote:
>
> Why I ask. It is not only matter of depending on the latest version, but for
> AWS, in the specific case, it probably even causes the error.
>
> When EC2 instance is configured with instance profile, as described here,
> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html,
> so new credentials are issued and periodically are renewed.
>
> And this is the error we see in the log from time to time (when running
> Ignite in the clustered mode):
>
> /ERROR [tcp-client-disco-reconnector-#5%%] o.a.i.s.d.t.TcpDiscoverySpi
> [null] - Failed to get registered addresses from IP fi
> nder on start (retrying every 2000 ms).
> org.apache.ignite.spi.IgniteSpiException: Failed to list objects in the
> bucket:
> at
> org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:168)
> ~[ignite-aws-1.7.0.ja
> r!/:1.7.0]
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1600)
> ~[ignite-core-1.7.0.jar!/:1.7.0]
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1549)
> ~[ignite-core-1.7.0.jar!/:1.7.0]
> at
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:475)
> [ignite-core-1.7.0.jar!/:1.7.0]
> at
> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
> [ignite-core-1.7.0.jar!/:1.7.0]
> at
> org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1175)
> [ignite-core-1.7.0.jar!/:1.7.0]
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> [ignite-core-1.7.0.jar!/:1.7.0]
> Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The provided
> token has expired. (Service: Amazon S3; Status Code: 400; Error Code:
> ExpiredToken;
> Request ID: EFDFC0BD8F4421AA)
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1307)
> ~[aws-java-sdk-core-1.10.50.jar!/:na]
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:894)
> ~[aws-java-sdk-core-1.10.50.jar!/:na]
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:597)
> ~[aws-java-sdk-core-1.10.50.jar!/:na]
> at
> com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:363)
> ~[aws-java-sdk-core-1.10.50.jar!/:na]
> at
> com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:329)
> ~[aws-java-sdk-core-1.10.50.jar!/:na]
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:308)
> ~[aws-java-sdk-core-1.10.50.jar!/:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
> ~[aws-java-sdk-s3-1.10.29.jar!/:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3548)
> ~[aws-java-sdk-s3-1.10.29.jar!/:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:647)
> ~[aws-java-sdk-s3-1.10.29.jar!/:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:626)
> ~[aws-java-sdk-s3-1.10.29.jar!/:na]
> at
> org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:128)
> ~[ignite-aws-1.7.0.ja
> r!/:1.7.0]
> ... 6 common frames omitted/
>
> Which probably could be due to S3 library is not being worked properly with
> EC2 instance profile. So, using the latest S3 AWS SDK, would be a good thing
> to try, whether this indeed was fixed in the new version.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9855.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by zshamrock <al...@gmail.com>.
Why I ask. It is not only matter of depending on the latest version, but for
AWS, in the specific case, it probably even causes the error.
When EC2 instance is configured with instance profile, as described here,
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html,
so new credentials are issued and periodically are renewed.
And this is the error we see in the log from time to time (when running
Ignite in the clustered mode):
/ERROR [tcp-client-disco-reconnector-#5%%] o.a.i.s.d.t.TcpDiscoverySpi
[null] - Failed to get registered addresses from IP fi
nder on start (retrying every 2000 ms).
org.apache.ignite.spi.IgniteSpiException: Failed to list objects in the
bucket:
at
org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:168)
~[ignite-aws-1.7.0.ja
r!/:1.7.0]
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1600)
~[ignite-core-1.7.0.jar!/:1.7.0]
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1549)
~[ignite-core-1.7.0.jar!/:1.7.0]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:475)
[ignite-core-1.7.0.jar!/:1.7.0]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
[ignite-core-1.7.0.jar!/:1.7.0]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1175)
[ignite-core-1.7.0.jar!/:1.7.0]
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[ignite-core-1.7.0.jar!/:1.7.0]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The provided
token has expired. (Service: Amazon S3; Status Code: 400; Error Code:
ExpiredToken;
Request ID: EFDFC0BD8F4421AA)
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1307)
~[aws-java-sdk-core-1.10.50.jar!/:na]
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:894)
~[aws-java-sdk-core-1.10.50.jar!/:na]
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:597)
~[aws-java-sdk-core-1.10.50.jar!/:na]
at
com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:363)
~[aws-java-sdk-core-1.10.50.jar!/:na]
at
com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:329)
~[aws-java-sdk-core-1.10.50.jar!/:na]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:308)
~[aws-java-sdk-core-1.10.50.jar!/:na]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
~[aws-java-sdk-s3-1.10.29.jar!/:na]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3548)
~[aws-java-sdk-s3-1.10.29.jar!/:na]
at
com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:647)
~[aws-java-sdk-s3-1.10.29.jar!/:na]
at
com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:626)
~[aws-java-sdk-s3-1.10.29.jar!/:na]
at
org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:128)
~[ignite-aws-1.7.0.ja
r!/:1.7.0]
... 6 common frames omitted/
Which probably could be due to S3 library is not being worked properly with
EC2 instance profile. So, using the latest S3 AWS SDK, would be a good thing
to try, whether this indeed was fixed in the new version.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9855.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by Denis Magda <dm...@apache.org>.
CC-ing dev@ list.
Created the ticket to update all 3rd party dependencies for our cloud based IP finders (AWS, GCE and JClouds).
https://issues.apache.org/jira/browse/IGNITE-4519 <https://issues.apache.org/jira/browse/IGNITE-4519>
Is there anyone in community who are interested in handling this?
—
Denis
> On Jan 2, 2017, at 12:11 PM, vkulichenko <va...@gmail.com> wrote:
>
> This is just a version ignite-aws module was developed on, and I don't think
> newer versions were ever tested. However, I believe it should work with
> 1.11.76. Can you try it out and let us know if it does?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9825.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by Denis Magda <dm...@apache.org>.
CC-ing dev@ list.
Created the ticket to update all 3rd party dependencies for our cloud based IP finders (AWS, GCE and JClouds).
https://issues.apache.org/jira/browse/IGNITE-4519 <https://issues.apache.org/jira/browse/IGNITE-4519>
Is there anyone in community who are interested in handling this?
—
Denis
> On Jan 2, 2017, at 12:11 PM, vkulichenko <va...@gmail.com> wrote:
>
> This is just a version ignite-aws module was developed on, and I don't think
> newer versions were ever tested. However, I believe it should work with
> 1.11.76. Can you try it out and let us know if it does?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9825.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Old AWS SDK version, why?
Posted by vkulichenko <va...@gmail.com>.
This is just a version ignite-aws module was developed on, and I don't think
newer versions were ever tested. However, I believe it should work with
1.11.76. Can you try it out and let us know if it does?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Old-AWS-SDK-version-why-tp9824p9825.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.