You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Adrian Cole (JIRA)" <ji...@apache.org> on 2011/08/13 13:08:27 UTC
[jira] [Created] (WHIRR-362) BlobStore contexts are redundantly
created
BlobStore contexts are redundantly created
------------------------------------------
Key: WHIRR-362
URL: https://issues.apache.org/jira/browse/WHIRR-362
Project: Whirr
Issue Type: Improvement
Components: cli
Affects Versions: 0.6.0
Reporter: Adrian Cole
Assignee: Adrian Cole
Fix For: 0.6.0
Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-362) BlobStore contexts are redundantly
created
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084610#comment-13084610 ]
Adrian Cole commented on WHIRR-362:
-----------------------------------
new patch tested against the ECC walrus endpoint:
whirr.blobstore-provider=walrus
whirr.blobstore-identity=XXXX
whirr.blobstore-credential=XXXX
jclouds.walrus.endpoint=http://partnercloud.eucalyptus.com:8773/services/Walrus
jclouds.walrus.iso3166-codes=US-CA
2011-08-13 14:14:23,568 INFO [org.apache.whirr.service.BlobStoreContextBuilder$Cache] (main) created new BlobStoreContext [id=walrus, endpoint=http://partnercloud.eucalyptus.com:8773/services/Walrus, apiVersion=Walrus-1.6, identity=5vOVwiDWcArl0zzqDLlaQPXdIhynAxz9ZSLapw, iso3166Codes=[US-CA]]
2011-08-13 14:14:26,111 INFO [org.apache.whirr.util.BlobCache] (main) Created blob cache container '3kq6ii50pfaz' located in 'null'
2011-08-13 14:14:27,463 INFO [org.apache.whirr.util.BlobCache] (main) Uploading 'whirr7960498924063392388.txt' to '3kq6ii50pfaz' blob cache.
2011-08-13 14:14:32,950 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) (mkdir -p /tmp && cd /tmp && [ ! -f whirr7960498924063392388.txt ] && curl -C - -s -q -L --connect-timeout 10 --max-time 600 -X GET -s --retry 20 -H "Date: Sat, 13 Aug 2011 13:14:23 GMT" -H "Authorization: AWS 5vOVwiDWcArl0zzqDLlaQPXdIhynAxz9ZSLapw:frDEUvWM6FUAzGOelP8iEOBqLwQ=" http://partnercloud.eucalyptus.com:8773/services/Walrus/3kq6ii50pfaz/whirr7960498924063392388.txt >whirr7960498924063392388.txt)
2011-08-13 14:14:32,950 INFO [org.apache.whirr.util.BlobCache] (main) Removing blob cache '3kq6ii50pfaz'
2011-08-13 14:14:34,659 INFO [org.apache.whirr.util.BlobCache] (main) Created blob cache container 'jhp3iaisleos' located in 'null'
2011-08-13 14:14:34,906 INFO [org.apache.whirr.util.BlobCache] (main) Uploading 'whirr6631307362110065793.txt' to 'jhp3iaisleos' blob cache.
2011-08-13 14:15:15,087 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) (mkdir -p /tmp && cd /tmp && [ ! -f whirr6631307362110065793.txt ] && curl -C - -s -q -L --connect-timeout 10 --max-time 600 -X GET -s --retry 20 -H "Date: Sat, 13 Aug 2011 13:14:23 GMT" -H "Authorization: AWS 5vOVwiDWcArl0zzqDLlaQPXdIhynAxz9ZSLapw:vr5UFrrd7LcsqNRxWfB2RqoFOho=" http://partnercloud.eucalyptus.com:8773/services/Walrus/jhp3iaisleos/whirr6631307362110065793.txt >whirr6631307362110065793.txt)
2011-08-13 14:15:15,088 INFO [org.apache.whirr.util.BlobCache] (main) Removing blob cache 'jhp3iaisleos'
2011-08-13 14:15:25,815 INFO [org.apache.whirr.service.ComputeCache] (main) created new ComputeServiceContext [id=aws-ec2, endpoint=https://ec2.us-east-1.amazonaws.com, apiVersion=2010-06-15, identity=067PW7Z9P0FNH7JDPE82, iso3166Codes=[US-VA, US-CA, IE, SG, JP-13]]
2011-08-13 14:15:26,505 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) Created temporary container 'h1lzpns2x5rr'
2011-08-13 14:15:26,769 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) Removing temporary container 'h1lzpns2x5rr'
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
> Attachments: BlobStoreContextBuilder.java, BlobStoreContextBuilder.java
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-362) BlobStore contexts are redundantly
created
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Cole updated WHIRR-362:
------------------------------
Attachment: BlobStoreContextBuilder.java
new version that properly handles user-configured jclouds providers like walrus
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
> Attachments: BlobStoreContextBuilder.java, BlobStoreContextBuilder.java
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-362) BlobStore contexts are redundantly
created
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084591#comment-13084591 ]
Adrian Cole commented on WHIRR-362:
-----------------------------------
Note this also allows you to pass context properties to BlobStore, something not supported prior. For example, the following configuration properly configures s3 to the eu region:
whirr.blobstore-location-id=EU
jclouds.aws-s3.endpoint=https://s3-eu-west-1.amazonaws.com
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
> Attachments: BlobStoreContextBuilder.java
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-362) BlobStore contexts are redundantly
created
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Cole updated WHIRR-362:
------------------------------
Attachment: BlobStoreContextBuilder.java
save as the following once WHIRR-361 is applied
./core/src/main/java/org/apache/whirr/service/BlobStoreContextBuilder.java
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
> Attachments: BlobStoreContextBuilder.java
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (WHIRR-362) BlobStore contexts are
redundantly created
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on WHIRR-362 started by Adrian Cole.
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-362) BlobStore contexts are redundantly
created
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Cole updated WHIRR-362:
------------------------------
Component/s: (was: cli)
core
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-362) BlobStore contexts are redundantly
created
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-362:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've just committed this.
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
> Attachments: BlobStoreContextBuilder.java, BlobStoreContextBuilder.java, WHIRR-362.patch
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-362) BlobStore contexts are redundantly
created
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Cole updated WHIRR-362:
------------------------------
Status: Patch Available (was: In Progress)
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
> Attachments: BlobStoreContextBuilder.java
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-362) BlobStore contexts are redundantly
created
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-362:
------------------------------
Attachment: WHIRR-362.patch
Created patch for this change. Tested on S3 & cloudfiles. The speed improvement is clearly visible.
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
> Attachments: BlobStoreContextBuilder.java, BlobStoreContextBuilder.java, WHIRR-362.patch
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-362) BlobStore contexts are redundantly
created
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084587#comment-13084587 ]
Adrian Cole commented on WHIRR-362:
-----------------------------------
tested, and noted log output where only a single BlobStoreContext is created during BlobCacheTest
> BlobStore contexts are redundantly created
> ------------------------------------------
>
> Key: WHIRR-362
> URL: https://issues.apache.org/jira/browse/WHIRR-362
> Project: Whirr
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6.0
> Reporter: Adrian Cole
> Assignee: Adrian Cole
> Fix For: 0.6.0
>
> Attachments: BlobStoreContextBuilder.java
>
>
> Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira