You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Chip Childers <ch...@sungard.com> on 2013/05/14 22:56:55 UTC
[ACS41] Next release candidate coming in a few hours!
Hi all,
We have a clear bug list (blockers and critical) for 4.1.0. I'm going
to cut a new release candidate tonight.
If there are *any* outstanding issues known, now's the time to raise
them.
(I'm specifically looking for an ACK from jburwell here, since he
mentioned a possible S3 feature issue. John, if you can please give me
a heads up on where you stand in the next few hours, I'd appreciate it.)
-chip
Re: [ACS41] Next release candidate coming in a few hours!
Posted by Sebastien Goasguen <ru...@gmail.com>.
On May 14, 2013, at 7:37 PM, Chip Childers <ch...@sungard.com> wrote:
> On Tuesday, May 14, 2013, John Burwell wrote:
>
>> Chip,
>>
>> After some further discussion about this issue on IRC, Alex and I
>> determined that system VM clock drift issue not only breaks S3, but has
>> other significant impacts that merit it being a blocker for 4.1 (e.g.
>> timestamps of files written by the SSVM being incorrect, log file
>> correlation difficulties, sensitivity of other storage protocols to time
>> sync, etc). Based on this conversation, I have opened CLOUDSTACK-2492<https://issues.apache.org/jira/browse/CLOUDSTACK-2492> to
>> capture the issue and track resolution.
>>
>> Thanks,
>> -John
>>
>
> Well shoot. Issuing a new system VM is going to require a bunch of testing
> if it's not done as a hand rolled edit of the existing system VMs.
>
> Who can take this?
Was this in 4.0.x ?
If yes then maybe we can live with it till 4.2 ?
>
>
>>
>> On May 14, 2013, at 6:09 PM, John Burwell <jburwell@basho.com<javascript:_e({}, 'cvml', 'jburwell@basho.com');>>
>> wrote:
>>
>> Chip,
>>
>> The source of the problem appears to be clock drift between the SSVM and
>> S3 per following stack trace:
>>
>> 2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils]
>> (agentRequest-Handler-3:) Putting directory
>> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in
>> S3 bucket jsb-cloudstack-templates.
>> 2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils]
>> (agentRequest-Handler-3:) Creating S3 client with configuration: [protocol:
>> https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000]
>> 2013-05-14 06:51:55,403 DEBUG
>> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
>> Determining key using account id 1 and template id 5
>> 2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils]
>> (agentRequest-Handler-3:) Putting file
>> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties
>> into bucket jsb-cloudstack-templates with key
>> template/tmpl/1/5/template.properties.
>> 2013-05-14 06:51:55,578 ERROR
>> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
>> Failed to upload template id 5
>> Status Code: 403, AWS Service: Amazon S3, AWS Request ID:
>> 970A274E132A9ACB, AWS Error Code: RequestTimeTooSkewed, AWS Error Message:
>> The difference between the request time and the current time is too large.,
>> S3 Extended Request ID:
>> 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
>> at
>> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
>> at
>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
>> at
>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
>> at
>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
>> at
>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
>> at
>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
>> at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
>> at
>> com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
>> at
>> com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
>> at com.cloud.agent.Agent.processRequest(Agent.java:525)
>> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>> at com.cloud.utils.nio.Task.run(Task.java:83)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> It does not appear that the System VM image has NTP nor does it seem like
>> a valid assumption that an SSVM would have the connectivity to reach an NTP
>> server. Therefore, do we have some other means to sync the clock on the
>> SSVM (e.g. with the host through hypervisor extensions)? In my previous
>> tests, the SSVM seemed to be syncing its time to the host which addressed
>> the problem.
>>
>> Thanks,
>> -John
>>
>> On May 14, 2013, at 4:58 PM, John Burwell <jburwell@basho.com<javascript:_e({}, 'cvml', 'jburwell@basho.com');>>
>> wrote:
>>
>> Chip,
>>
>> I am looking into the issue now. There is a failure when the S3 upload
>> template command is issued. I working to determine whether or not the
>> cause is environmental or code.
>>
>> Thanks,
>> -John
>>
>> On May 14, 2013, at 4:56 PM, Chip Childers <chip.childers@sungard.com<javascript:_e({}, 'cvml', 'chip.childers@sungard.com');>>
>> wrote:
>>
>> Hi all,
>>
>> We have a clear bug list (blockers and critical) for 4.1.0. I'm going
>> to cut a new release candidate tonight.
>>
>> If there are *any* outstanding issues known, now's the time to raise
>> them.
>>
>> (I'm specifically looking for an ACK from jburwell here, since he
>> mentioned a possible S3 feature issue. John, if you can please give me
>> a heads up on where you stand in the next few hours, I'd appreciate it.)
>>
>> -chip
>>
>>
>>
>>
>>
Re: [ACS41] Next release candidate coming in a few hours!
Posted by Chip Childers <ch...@sungard.com>.
On Tuesday, May 14, 2013, John Burwell wrote:
> Chip,
>
> After some further discussion about this issue on IRC, Alex and I
> determined that system VM clock drift issue not only breaks S3, but has
> other significant impacts that merit it being a blocker for 4.1 (e.g.
> timestamps of files written by the SSVM being incorrect, log file
> correlation difficulties, sensitivity of other storage protocols to time
> sync, etc). Based on this conversation, I have opened CLOUDSTACK-2492<https://issues.apache.org/jira/browse/CLOUDSTACK-2492> to
> capture the issue and track resolution.
>
> Thanks,
> -John
>
Well shoot. Issuing a new system VM is going to require a bunch of testing
if it's not done as a hand rolled edit of the existing system VMs.
Who can take this?
>
> On May 14, 2013, at 6:09 PM, John Burwell <jburwell@basho.com<javascript:_e({}, 'cvml', 'jburwell@basho.com');>>
> wrote:
>
> Chip,
>
> The source of the problem appears to be clock drift between the SSVM and
> S3 per following stack trace:
>
> 2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils]
> (agentRequest-Handler-3:) Putting directory
> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in
> S3 bucket jsb-cloudstack-templates.
> 2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils]
> (agentRequest-Handler-3:) Creating S3 client with configuration: [protocol:
> https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000]
> 2013-05-14 06:51:55,403 DEBUG
> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
> Determining key using account id 1 and template id 5
> 2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils]
> (agentRequest-Handler-3:) Putting file
> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties
> into bucket jsb-cloudstack-templates with key
> template/tmpl/1/5/template.properties.
> 2013-05-14 06:51:55,578 ERROR
> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
> Failed to upload template id 5
> Status Code: 403, AWS Service: Amazon S3, AWS Request ID:
> 970A274E132A9ACB, AWS Error Code: RequestTimeTooSkewed, AWS Error Message:
> The difference between the request time and the current time is too large.,
> S3 Extended Request ID:
> 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
> at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
> at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
> at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
> at
> com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
> at
> com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
> at com.cloud.agent.Agent.processRequest(Agent.java:525)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> It does not appear that the System VM image has NTP nor does it seem like
> a valid assumption that an SSVM would have the connectivity to reach an NTP
> server. Therefore, do we have some other means to sync the clock on the
> SSVM (e.g. with the host through hypervisor extensions)? In my previous
> tests, the SSVM seemed to be syncing its time to the host which addressed
> the problem.
>
> Thanks,
> -John
>
> On May 14, 2013, at 4:58 PM, John Burwell <jburwell@basho.com<javascript:_e({}, 'cvml', 'jburwell@basho.com');>>
> wrote:
>
> Chip,
>
> I am looking into the issue now. There is a failure when the S3 upload
> template command is issued. I working to determine whether or not the
> cause is environmental or code.
>
> Thanks,
> -John
>
> On May 14, 2013, at 4:56 PM, Chip Childers <chip.childers@sungard.com<javascript:_e({}, 'cvml', 'chip.childers@sungard.com');>>
> wrote:
>
> Hi all,
>
> We have a clear bug list (blockers and critical) for 4.1.0. I'm going
> to cut a new release candidate tonight.
>
> If there are *any* outstanding issues known, now's the time to raise
> them.
>
> (I'm specifically looking for an ACK from jburwell here, since he
> mentioned a possible S3 feature issue. John, if you can please give me
> a heads up on where you stand in the next few hours, I'd appreciate it.)
>
> -chip
>
>
>
>
>
Re: [ACS41] Next release candidate coming in a few hours!
Posted by John Burwell <jb...@basho.com>.
Chip,
After some further discussion about this issue on IRC, Alex and I determined that system VM clock drift issue not only breaks S3, but has other significant impacts that merit it being a blocker for 4.1 (e.g. timestamps of files written by the SSVM being incorrect, log file correlation difficulties, sensitivity of other storage protocols to time sync, etc). Based on this conversation, I have opened CLOUDSTACK-2492 to capture the issue and track resolution.
Thanks,
-John
On May 14, 2013, at 6:09 PM, John Burwell <jb...@basho.com> wrote:
> Chip,
>
> The source of the problem appears to be clock drift between the SSVM and S3 per following stack trace:
>
> 2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Putting directory /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in S3 bucket jsb-cloudstack-templates.
> 2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Creating S3 client with configuration: [protocol: https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000]
> 2013-05-14 06:51:55,403 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Determining key using account id 1 and template id 5
> 2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Putting file /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties into bucket jsb-cloudstack-templates with key template/tmpl/1/5/template.properties.
> 2013-05-14 06:51:55,578 ERROR [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Failed to upload template id 5
> Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 970A274E132A9ACB, AWS Error Code: RequestTimeTooSkewed, AWS Error Message: The difference between the request time and the current time is too large., S3 Extended Request ID: 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
> at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
> at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
> at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
> at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
> at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
> at com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
> at com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
> at com.cloud.agent.Agent.processRequest(Agent.java:525)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> It does not appear that the System VM image has NTP nor does it seem like a valid assumption that an SSVM would have the connectivity to reach an NTP server. Therefore, do we have some other means to sync the clock on the SSVM (e.g. with the host through hypervisor extensions)? In my previous tests, the SSVM seemed to be syncing its time to the host which addressed the problem.
>
> Thanks,
> -John
>
> On May 14, 2013, at 4:58 PM, John Burwell <jb...@basho.com> wrote:
>
>> Chip,
>>
>> I am looking into the issue now. There is a failure when the S3 upload template command is issued. I working to determine whether or not the cause is environmental or code.
>>
>> Thanks,
>> -John
>>
>> On May 14, 2013, at 4:56 PM, Chip Childers <ch...@sungard.com> wrote:
>>
>>> Hi all,
>>>
>>> We have a clear bug list (blockers and critical) for 4.1.0. I'm going
>>> to cut a new release candidate tonight.
>>>
>>> If there are *any* outstanding issues known, now's the time to raise
>>> them.
>>>
>>> (I'm specifically looking for an ACK from jburwell here, since he
>>> mentioned a possible S3 feature issue. John, if you can please give me
>>> a heads up on where you stand in the next few hours, I'd appreciate it.)
>>>
>>> -chip
>>
>
Re: [ACS41] Next release candidate coming in a few hours!
Posted by John Burwell <jb...@basho.com>.
Chip,
The source of the problem appears to be clock drift between the SSVM and S3 per following stack trace:
2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Putting directory /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in S3 bucket jsb-cloudstack-templates.
2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Creating S3 client with configuration: [protocol: https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000]
2013-05-14 06:51:55,403 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Determining key using account id 1 and template id 5
2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Putting file /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties into bucket jsb-cloudstack-templates with key template/tmpl/1/5/template.properties.
2013-05-14 06:51:55,578 ERROR [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Failed to upload template id 5
Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 970A274E132A9ACB, AWS Error Code: RequestTimeTooSkewed, AWS Error Message: The difference between the request time and the current time is too large., S3 Extended Request ID: 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
at com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
at com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
at com.cloud.utils.nio.Task.run(Task.java:83)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
It does not appear that the System VM image has NTP nor does it seem like a valid assumption that an SSVM would have the connectivity to reach an NTP server. Therefore, do we have some other means to sync the clock on the SSVM (e.g. with the host through hypervisor extensions)? In my previous tests, the SSVM seemed to be syncing its time to the host which addressed the problem.
Thanks,
-John
On May 14, 2013, at 4:58 PM, John Burwell <jb...@basho.com> wrote:
> Chip,
>
> I am looking into the issue now. There is a failure when the S3 upload template command is issued. I working to determine whether or not the cause is environmental or code.
>
> Thanks,
> -John
>
> On May 14, 2013, at 4:56 PM, Chip Childers <ch...@sungard.com> wrote:
>
>> Hi all,
>>
>> We have a clear bug list (blockers and critical) for 4.1.0. I'm going
>> to cut a new release candidate tonight.
>>
>> If there are *any* outstanding issues known, now's the time to raise
>> them.
>>
>> (I'm specifically looking for an ACK from jburwell here, since he
>> mentioned a possible S3 feature issue. John, if you can please give me
>> a heads up on where you stand in the next few hours, I'd appreciate it.)
>>
>> -chip
>
Re: [ACS41] Next release candidate coming in a few hours!
Posted by John Burwell <jb...@basho.com>.
Chip,
I am looking into the issue now. There is a failure when the S3 upload template command is issued. I working to determine whether or not the cause is environmental or code.
Thanks,
-John
On May 14, 2013, at 4:56 PM, Chip Childers <ch...@sungard.com> wrote:
> Hi all,
>
> We have a clear bug list (blockers and critical) for 4.1.0. I'm going
> to cut a new release candidate tonight.
>
> If there are *any* outstanding issues known, now's the time to raise
> them.
>
> (I'm specifically looking for an ACK from jburwell here, since he
> mentioned a possible S3 feature issue. John, if you can please give me
> a heads up on where you stand in the next few hours, I'd appreciate it.)
>
> -chip