You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jclouds.apache.org by Peter Halliday <pe...@datastax.com> on 2014/12/05 17:41:48 UTC
Thread being killed
I’m using JClouds BlobStore in an application that’s using it to send files to S3. It does this in batches like after a snapshot is run on a database. And then all of the files are transferred to S3. We were testing error states. And we were testing changing a User’s Access credentials from inactive to active and back. When we go from active to inactive credentials during this transfer to S3. Our thread gets killed. The last log entry is from JClouds. Our code is literally catching all errors, and so nothing is being thrown at that time. Our application is in clojure. You can see in the log, that we upload successfully fine, and then after we switch to invalid we “correct” get errors that get returned to our application. However, then we hit the SEVERE log line, and then our thread is dead and never comes back :-(
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,398 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Data.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,496 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-CompressionInfo.db
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,498 Throttle value requested is out of range, using default.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,695 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-CompressionInfo.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,797 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-Index.db
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,809 Throttle value requested is out of range, using default.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,105 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,204 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Statistics.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,305 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,406 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,503 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@48fee98c to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,504 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Summary.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,600 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@91afeb42 to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,601 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Filter.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Filter.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,696 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@a4c93 to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,697 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-CompressionInfo.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-CompressionInfo.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,792 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@44e5d97 to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,793 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Statistics.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Statistics.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,888 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@8ca4473b to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,890 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Data.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Data.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
DEBUG [node-details-1] 2014-12-05 16:29:16,030 Starting real-time collection
DEBUG [node-details-1] 2014-12-05 16:29:16,030 Collecting real-time data
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,034 Sending message '["172.31.10.206",{"os-load":0.07}]' to channel '/node-details'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,298 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"schema","state":"running","details":{"keyspace":"OpsCenter","filename":"OpsCenter\/schema.json"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
DEBUG [async-dispatch-31] 2014-12-05 16:29:16,402 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
DEBUG [async-dispatch-31] 2014-12-05 16:29:16,404 Throttle value requested is out of range, using default.
ERROR [async-dispatch-31] 2014-12-05 16:29:16,510 Dec 05, 2014 4:29:16 PM org.jclouds.logging.jdk.JDKLogger logError
SEVERE: Cannot retry after server error, command is not replayable: [method=org.jclouds.aws.s3.AWSS3AsyncClient.public abstract com.google.common.util.concurrent.ListenableFuture org.jclouds.s3.S3AsyncClient.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[phalliday-opscenter-backups-4, [metadata=[key=snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db, bucket=null, uri=null, eTag=null, cacheControl=null, contentMetadata=[contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=107, contentMD5=null, contentType=application/octet-stream, expires=null], lastModified=null, owner=null, storageClass=STANDARD, userMetadata={}]], [Lorg.jclouds.s3.options.PutObjectOptions;@4e0ecbfe], request=PUT https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db HTTP/1.1]
DEBUG [node-details-1] 2014-12-05 16:29:21,034 Starting real-time collection
DEBUG [node-details-1] 2014-12-05 16:29:21,035 Collecting real-time data
DEBUG [node-details-1] 2014-12-05 16:29:21,038 Send requested, but nothing new to send
<http://www.datastax.com/>
PETER HALLIDAY
OpsCenter | Sr Software Engineer | 607.329.6905 | peter@datastax.com
<https://twitter.com/phalliday> <https://www.linkedin.com/profile/view?id=7030852>
Re: Thread being killed
Posted by Peter Halliday <pe...@datastax.com>.
My alarm has nothing to do with the word SEVERE, it’s more due to the fact that my thread dies. I’m just using the blobstore clojure api, I’m not familiar with the async vs sync denotation in coordination with that API.
<http://www.datastax.com/>
PETER HALLIDAY
OpsCenter | Sr Software Engineer | 607.329.6905 | peter@datastax.com <ma...@datastax.com>
<https://twitter.com/phalliday> <https://www.linkedin.com/profile/view?id=7030852>
> On Dec 5, 2014, at 11:12 AM, Andrew Gaul <gaul@apache.org <ma...@apache.org>> wrote:
>
> [Moving from the dev to the user mailing list.]
>
> The SEVERE log message is not as alarming as you might suspect; JDK
> logger maps error level logs to severe level. It appears that you are
> using the deprecated async jclouds interface instead of the sync
> interface; are you consuming the error response via Future.get?
> Otherwise, can you minimize a test case so we can diagnose this further?
>
> On Fri, Dec 05, 2014 at 10:41:48AM -0600, Peter Halliday wrote:
>> I’m using JClouds BlobStore in an application that’s using it to send files to S3. It does this in batches like after a snapshot is run on a database. And then all of the files are transferred to S3. We were testing error states. And we were testing changing a User’s Access credentials from inactive to active and back. When we go from active to inactive credentials during this transfer to S3. Our thread gets killed. The last log entry is from JClouds. Our code is literally catching all errors, and so nothing is being thrown at that time. Our application is in clojure. You can see in the log, that we upload successfully fine, and then after we switch to invalid we “correct” get errors that get returned to our application. However, then we hit the SEVERE log line, and then our thread is dead and never comes back :-(
>>
>>
>>
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,398 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Data.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,496 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-CompressionInfo.db
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,498 Throttle value requested is out of range, using default.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,695 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-CompressionInfo.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,797 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-Index.db
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,809 Throttle value requested is out of range, using default.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,105 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,204 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Statistics.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,305 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,406 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,503 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@48fee98c to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> ... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,504 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Summary.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,600 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@91afeb42 to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> ... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,601 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Filter.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Filter.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,696 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@a4c93 to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> ... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,697 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-CompressionInfo.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-CompressionInfo.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,792 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@44e5d97 to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> ... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,793 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Statistics.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Statistics.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,888 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@8ca4473b to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db> HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db> HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> ... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,890 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Data.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Data.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Starting real-time collection
>> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Collecting real-time data
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,034 Sending message '["172.31.10.206",{"os-load":0.07}]' to channel '/node-details'.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,298 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"schema","state":"running","details":{"keyspace":"OpsCenter","filename":"OpsCenter\/schema.json"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,402 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,404 Throttle value requested is out of range, using default.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:16,510 Dec 05, 2014 4:29:16 PM org.jclouds.logging.jdk.JDKLogger logError
>> SEVERE: Cannot retry after server error, command is not replayable: [method=org.jclouds.aws.s3.AWSS3AsyncClient.public abstract com.google.common.util.concurrent.ListenableFuture org.jclouds.s3.S3AsyncClient.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[phalliday-opscenter-backups-4, [metadata=[key=snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db, bucket=null, uri=null, eTag=null, cacheControl=null, contentMetadata=[contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=107, contentMD5=null, contentType=application/octet-stream, expires=null], lastModified=null, owner=null, storageClass=STANDARD, userMetadata={}]], [Lorg.jclouds.s3.options.PutObjectOptions;@4e0ecbfe], request=PUT https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db <https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db> HTTP/1.1]
>>
>> DEBUG [node-details-1] 2014-12-05 16:29:21,034 Starting real-time collection
>> DEBUG [node-details-1] 2014-12-05 16:29:21,035 Collecting real-time data
>> DEBUG [node-details-1] 2014-12-05 16:29:21,038 Send requested, but nothing new to send
>>
>>
>>
>>
>>
>> <http://www.datastax.com/ <http://www.datastax.com/>>
>> PETER HALLIDAY
>> OpsCenter | Sr Software Engineer | 607.329.6905 | peter@datastax.com <ma...@datastax.com>
>>
>> <https://twitter.com/phalliday <https://twitter.com/phalliday>> <https://www.linkedin.com/profile/view?id=7030852 <https://www.linkedin.com/profile/view?id=7030852>>
>
> --
> Andrew Gaul
> http://gaul.org/ <http://gaul.org/>
Re: Thread being killed
Posted by Andrew Gaul <ga...@apache.org>.
[Moving from the dev to the user mailing list.]
The SEVERE log message is not as alarming as you might suspect; JDK
logger maps error level logs to severe level. It appears that you are
using the deprecated async jclouds interface instead of the sync
interface; are you consuming the error response via Future.get?
Otherwise, can you minimize a test case so we can diagnose this further?
On Fri, Dec 05, 2014 at 10:41:48AM -0600, Peter Halliday wrote:
> I’m using JClouds BlobStore in an application that’s using it to send files to S3. It does this in batches like after a snapshot is run on a database. And then all of the files are transferred to S3. We were testing error states. And we were testing changing a User’s Access credentials from inactive to active and back. When we go from active to inactive credentials during this transfer to S3. Our thread gets killed. The last log entry is from JClouds. Our code is literally catching all errors, and so nothing is being thrown at that time. Our application is in clojure. You can see in the log, that we upload successfully fine, and then after we switch to invalid we “correct” get errors that get returned to our application. However, then we hit the SEVERE log line, and then our thread is dead and never comes back :-(
>
>
>
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,398 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Data.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,496 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-CompressionInfo.db
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,498 Throttle value requested is out of range, using default.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,695 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-CompressionInfo.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,797 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-Index.db
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,809 Throttle value requested is out of range, using default.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,105 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,204 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Statistics.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,305 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,406 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,503 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@48fee98c to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,504 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Summary.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,600 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@91afeb42 to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,601 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Filter.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Filter.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,696 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@a4c93 to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,697 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-CompressionInfo.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-CompressionInfo.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,792 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@44e5d97 to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,793 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Statistics.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Statistics.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,888 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@8ca4473b to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db HTTP/1.1 -> HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> ... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,890 Sending message '["172.31.10.206",{"details":{"error":"HEAD https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Data.db HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Data.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Starting real-time collection
> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Collecting real-time data
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,034 Sending message '["172.31.10.206",{"os-load":0.07}]' to channel '/node-details'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,298 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"schema","state":"running","details":{"keyspace":"OpsCenter","filename":"OpsCenter\/schema.json"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]' to channel '/snapshot/status'.
> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,402 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,404 Throttle value requested is out of range, using default.
> ERROR [async-dispatch-31] 2014-12-05 16:29:16,510 Dec 05, 2014 4:29:16 PM org.jclouds.logging.jdk.JDKLogger logError
> SEVERE: Cannot retry after server error, command is not replayable: [method=org.jclouds.aws.s3.AWSS3AsyncClient.public abstract com.google.common.util.concurrent.ListenableFuture org.jclouds.s3.S3AsyncClient.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[phalliday-opscenter-backups-4, [metadata=[key=snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db, bucket=null, uri=null, eTag=null, cacheControl=null, contentMetadata=[contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=107, contentMD5=null, contentType=application/octet-stream, expires=null], lastModified=null, owner=null, storageClass=STANDARD, userMetadata={}]], [Lorg.jclouds.s3.options.PutObjectOptions;@4e0ecbfe], request=PUT https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db HTTP/1.1]
>
> DEBUG [node-details-1] 2014-12-05 16:29:21,034 Starting real-time collection
> DEBUG [node-details-1] 2014-12-05 16:29:21,035 Collecting real-time data
> DEBUG [node-details-1] 2014-12-05 16:29:21,038 Send requested, but nothing new to send
>
>
>
>
>
> <http://www.datastax.com/>
> PETER HALLIDAY
> OpsCenter | Sr Software Engineer | 607.329.6905 | peter@datastax.com
>
> <https://twitter.com/phalliday> <https://www.linkedin.com/profile/view?id=7030852>
--
Andrew Gaul
http://gaul.org/