You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Nico Herzhauser <he...@hotmail.de> on 2015/07/14 14:58:15 UTC
Need help with S3 Secondary Storage
Hello cloudstack usergroup,
we like to try S3 style storage with cloudstack 4.5 but we cannot connect to the S3 Storage.We think this is a certificate problem because the ssvm did not get the right certificate.
We use a Wildcard SSL certificate.
At the storage-vm I see the following error in the log file:
2015-07-13 13:59:54,887 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 40-6480961338762854402: { Ans: , MgmtId: 345049465082, via: 40, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"details":"","wait":0}}] }2015-07-13 13:59:55,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Request:Seq 40-6480961338762854403: { Cmd , MgmtId: 345049465082, via: 40, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.ListVolumeCommand":{"store":{"com.cloud.agent.api.to.S3TO":{"id":15,"uuid":"51333282-9c81-43ca-9532-fac88f722df9","endPoint":"%fqdn%","bucketName":"secondary","httpsFlag":true,"created":"Jul 13, 2015 4:02:25 PM","enableRRS":false,"maxSingleUploadSizeInBytes":5368709120}},"wait":0}}] }2015-07-13 13:59:55,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Processing command: com.cloud.agent.api.storage.ListVolumeCommand2015-07-13 13:59:55,031 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:null) Creating S3 client with configuration: [protocol: https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000]2015-07-13 13:59:55,160 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:null) Setting the end point for S3 client com.amazonaws.services.s3.AmazonS3Client@6c05762a to %fqdn%.2015-07-13 13:59:55,549 INFO [amazonaws.http.AmazonHttpClient] (agentRequest-Handler-3:null) Unable to execute HTTP request: peer not authenticatedjavax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:421) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:278) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2878) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:452) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:436) at com.cloud.utils.S3Utils.listDirectory(S3Utils.java:341) at com.cloud.utils.S3Utils.getDirectory(S3Utils.java:336) at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.s3ListVolume(NfsSecondaryStorageResource.java:1432) at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:1486) at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:229) at com.cloud.storage.resource.PremiumSecondaryStorageResource.defaultAction(PremiumSecondaryStorageResource.java:64) at com.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:60) at com.cloud.agent.Agent.processRequest(Agent.java:506) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:814) at com.cloud.utils.nio.Task.run(Task.java:84) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)2015-07-13 13:59:56,226 INFO [amazonaws.http.AmazonHttpClient] (agentRequest-Handler-3:null) Unable to execute HTTP request: peer not authenticatedjavax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:421) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:278) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2878) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:452) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:436) at com.cloud.utils.S3Utils.listDirectory(S3Utils.java:341) at com.cloud.utils.S3Utils.getDirectory(S3Utils.java:336) at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.s3ListVolume(NfsSecondaryStorageResource.java:1432) at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:1486) at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:229) at com.cloud.storage.resource.PremiumSecondaryStorageResource.defaultAction(PremiumSecondaryStorageResource.java:64) at com.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:60) at com.cloud.agent.Agent.processRequest(Agent.java:506) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:814) at com.cloud.utils.nio.Task.run(Task.java:84) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
RE: Need help with S3 Secondary Storage
Posted by Nico Herzhauser <he...@hotmail.de>.
Hello dshevchenko,
we will try the workaround and we will give feedback if that worked for us or not.
> Date: Tue, 14 Jul 2015 16:46:23 +0300
> From: dshevchenko.mail@gmail.com
> To: users@cloudstack.apache.org
> Subject: Re: Need help with S3 Secondary Storage
>
> Hello Nico.
> We also trying to use S3 as secondary storage, so several thoughts:
> 1. "peer not authenticated" - maybe problem with access id and secret
> id? Can you authenticate with external client?
> 2. You cannot use self-signed certificate, it's not supported (actually
> you can, but it must be added as trusted to local java keystore on all
> nodes, including ssvm)
> 3. We also have problem with S3 via https and ssvm, because of ssvm
> using custom java keystore file (/realhostip.keystore/) and in this file
> only one trusted root certificate from godaddy.com. But even worse - in
> source code it hard-coded that you can inject your custom certificate to
> ssvm/cpvm (I mean trusted root cert here) only if secondary storage is NFS.
>
> As workaround: after installation download and unpack systemvm.iso, find
> realhostip.keystore file, add your trusted root or self-signed
> certificate into it via keytool utility, recreate new iso file and
> replace it on all management and KVM nodes.
>
> check this
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Procedure+to+Replace+realhostip.com+with+Your+Own+Domain+Name
> and
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Implementation+details+and+troubleshooting+-+uploading+custom+domain+certificate+instead+of+using+realhostip.com
>
> On 07/14/2015 03:58 PM, Nico Herzhauser wrote:
> > Hello cloudstack usergroup,
> > we like to try S3 style storage with cloudstack 4.5 but we cannot connect to the S3 Storage.We think this is a certificate problem because the ssvm did not get the right certificate.
> > We use a Wildcard SSL certificate.
> > At the storage-vm I see the following error in the log file:
> > 2015-07-13 13:59:54,887 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 40-6480961338762854402: { Ans: , MgmtId: 345049465082, via: 40, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"details":"","wait":0}}] }2015-07-13 13:59:55,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Request:Seq 40-6480961338762854403: { Cmd , MgmtId: 345049465082, via: 40, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.ListVolumeCommand":{"store":{"com.cloud.agent.api.to.S3TO":{"id":15,"uuid":"51333282-9c81-43ca-9532-fac88f722df9","endPoint":"%fqdn%","bucketName":"secondary","httpsFlag":true,"created":"Jul 13, 2015 4:02:25 PM","enableRRS":false,"maxSingleUploadSizeInBytes":5368709120}},"wait":0}}] }2015-07-13 13:59:55,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Processing command: com.cloud.agent.api.storage.ListVolumeCommand2015-07-13 13:59:55,031 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:null) Creating S3 client with configuration: [protocol: https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000]2015-07-13 13:59:55,160 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:null) Setting the end point for S3 client com.amazonaws.services.s3.AmazonS3Client@6c05762a to %fqdn%.2015-07-13 13:59:55,549 INFO [amazonaws.http.AmazonHttpClient] (agentRequest-Handler-3:null) Unable to execute HTTP request: peer not authenticatedjavax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:421) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at
>
Re: Need help with S3 Secondary Storage
Posted by Dmytro Shevchenko <ds...@gmail.com>.
Hello Nico.
We also trying to use S3 as secondary storage, so several thoughts:
1. "peer not authenticated" - maybe problem with access id and secret
id? Can you authenticate with external client?
2. You cannot use self-signed certificate, it's not supported (actually
you can, but it must be added as trusted to local java keystore on all
nodes, including ssvm)
3. We also have problem with S3 via https and ssvm, because of ssvm
using custom java keystore file (/realhostip.keystore/) and in this file
only one trusted root certificate from godaddy.com. But even worse - in
source code it hard-coded that you can inject your custom certificate to
ssvm/cpvm (I mean trusted root cert here) only if secondary storage is NFS.
As workaround: after installation download and unpack systemvm.iso, find
realhostip.keystore file, add your trusted root or self-signed
certificate into it via keytool utility, recreate new iso file and
replace it on all management and KVM nodes.
check this
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Procedure+to+Replace+realhostip.com+with+Your+Own+Domain+Name
and
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Implementation+details+and+troubleshooting+-+uploading+custom+domain+certificate+instead+of+using+realhostip.com
On 07/14/2015 03:58 PM, Nico Herzhauser wrote:
> Hello cloudstack usergroup,
> we like to try S3 style storage with cloudstack 4.5 but we cannot connect to the S3 Storage.We think this is a certificate problem because the ssvm did not get the right certificate.
> We use a Wildcard SSL certificate.
> At the storage-vm I see the following error in the log file:
> 2015-07-13 13:59:54,887 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 40-6480961338762854402: { Ans: , MgmtId: 345049465082, via: 40, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"details":"","wait":0}}] }2015-07-13 13:59:55,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Request:Seq 40-6480961338762854403: { Cmd , MgmtId: 345049465082, via: 40, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.ListVolumeCommand":{"store":{"com.cloud.agent.api.to.S3TO":{"id":15,"uuid":"51333282-9c81-43ca-9532-fac88f722df9","endPoint":"%fqdn%","bucketName":"secondary","httpsFlag":true,"created":"Jul 13, 2015 4:02:25 PM","enableRRS":false,"maxSingleUploadSizeInBytes":5368709120}},"wait":0}}] }2015-07-13 13:59:55,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Processing command: com.cloud.agent.api.storage.ListVolumeCommand2015-07-13 13:59:55,031 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:null) Creating S3 client with configuration: [protocol: https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000]2015-07-13 13:59:55,160 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:null) Setting the end point for S3 client com.amazonaws.services.s3.AmazonS3Client@6c05762a to %fqdn%.2015-07-13 13:59:55,549 INFO [amazonaws.http.AmazonHttpClient] (agentRequest-Handler-3:null) Unable to execute HTTP request: peer not authenticatedjavax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:421) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at