You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jclouds.apache.org by Shri Javadekar <no...@github.com> on 2013/12/18 07:42:24 UTC
[jclouds] Add BlobStoreBench to jclouds. (#239)
BlobStoreBench is simple utility to benchmark blobstore performance.
It uses the basic jclouds constructs to perform GET, PUT, DELETE
operations in a blobstore.
See the convenience script blobstorebench.sh to see how to run the
tool.
Tested READ/WRITE against atmos, aws-s3, azure, cloudfiles-us, s3
(api), hpcloud, filesystem, transient, swift-keystone.
You can merge this Pull Request by running:
git pull https://github.com/maginatics/jclouds intro-blobstorebench-v2
Or you can view, comment on it, or merge it online at:
https://github.com/jclouds/jclouds/pull/239
-- Commit Summary --
* Add BlobStoreBench to jclouds.
-- File Changes --
A blobstore-tools/blobstorebench.sh (58)
A blobstore-tools/pom.xml (77)
A blobstore-tools/src/main/java/org/jclouds/blobstore/tools/BlobStoreBench.java (637)
M pom.xml (1)
-- Patch Links --
https://github.com/jclouds/jclouds/pull/239.patch
https://github.com/jclouds/jclouds/pull/239.diff
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Gaul <no...@github.com>.
> +# Licensed to the Apache Software Foundation (ASF) under one or more
> +# contributor license agreements. See the NOTICE file distributed with
> +# this work for additional information regarding copyright ownership.
> +# The ASF licenses this file to You under the Apache License, Version 2.0
> +# (the "License"); you may not use this file except in compliance with
> +# the License. You may obtain a copy of the License at
> +#
> +# http://www.apache.org/licenses/LICENSE-2.0
> +#
> +# Unless required by applicable law or agreed to in writing, software
> +# distributed under the License is distributed on an "AS IS" BASIS,
> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> +# See the License for the specific language governing permissions and
> +# limitations under the License.
> +
> +#!/usr/bin/env bash
Shebangs only work in as the top-level comment. Also link to #!/bin/sh explicitly.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239/files#r8445761
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #947](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/947/) FAILURE
Looks like there's a problem with this pull request
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30820517
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #484](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/484/) FAILURE
Looks like there's a problem with this pull request
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30820489
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Phillips <no...@github.com>.
Agree on trying to keep the number of repos down. And I think @abayer might not be too unhappy about trying to at least combine karaf and cli, since in practice it seems like they version in lockstep anyway. Would like to hear from @iocanel on that subject, though.
My thinking behind jclouds-examples vs. jclouds-karaf/cli was that the size of this code is closer to the examples than to karaf or the cli, although from a functionality perspective I agree with you that this is more a utility than an example.
Is there perhaps some way this could be worked into a CLI command? Seems like a nice and powerful way to invoke this?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30862910
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Everett Toews <no...@github.com>.
During release week we like to do a little house cleaning in the jclouds world. That means sweeping out the pull request queue.
This PR is over 6 months old. Please update us on its status here. If we don't hear anything, we will take that as lazy consensus that the PR is no longer relevant and it will be closed on Friday, Aug 8.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-50782397
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Gaul <no...@github.com>.
> +# Blobstore credentials
> +IDENTITY="myidentity"
> +CREDENTIAL="secretCreds"
> +
> +# Container used for the test. For "transient" and "filesystem" providers,
> +# the container gets created at runtime. For others, it is expected to
> +# be present.
> +CONTAINER=jclouds-container
> +
> +# Start BlobStoreBench. Note that for some providers, you might have to add
> +# additional params below.
> +# For e.g.
> +# -Djclouds.keystone.credential-type=apiAccessKeyCredentials \
> +# -Djclouds.endpoint="http://some-endpoint.blobstore.com" \
> +# -Djclouds.strip-expect-header=true \
> +java -server \
Can we rewrite this with really-executable-jar in a more natural way?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239/files#r8445775
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Jeremy Daggett <no...@github.com>.
@demobox WDYT about creating a "jclouds-tools" repo for this kind of jclouds tooling? I don't think putting it in the examples project makes sense for these types of integrations. If we did this, we could reduce the number of repos by collapsing karaf/cli/chef into a single repo with subprojects. I don't know if this causes release/versioning issues with those projects, but it makes sense to me. :)
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30861948
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Phillips <no...@github.com>.
> I will investigate creating a jclouds-tools repository to host this commit and future tools.
Thanks, @andrewgaul!
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-38330578
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Phillips <no...@github.com>.
This looks related to the jclouds-tools discussion and would make a great addition to that repo, I think.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-50783161
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Zack Shoylev <no...@github.com>.
Perhaps examples is not the place for this, but can we do this without more repos? /cringe
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30862250
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Shri Javadekar <no...@github.com>.
> +import org.kohsuke.args4j.Option;
> +
> +import com.google.common.base.Preconditions;
> +import com.google.common.base.Stopwatch;
> +import com.google.common.collect.Iterables;
> +import com.google.common.collect.Lists;
> +import com.google.common.io.ByteStreams;
> +import com.google.common.net.MediaType;
> +import com.google.common.util.concurrent.FutureCallback;
> +import com.google.common.util.concurrent.Futures;
> +import com.google.common.util.concurrent.ListenableFuture;
> +import com.google.common.util.concurrent.ListeningExecutorService;
> +import com.google.common.util.concurrent.MoreExecutors;
> +
> +/**
> + * Simple BlobStore benchmark.
Done.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239/files#r8645475
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Gaul <no...@github.com>.
> + <shadedClassifierName>jar-with-dependencies</shadedClassifierName>
> + <createDependencyReducedPom>false</createDependencyReducedPom>
> + <shadedArtifactAttached>true</shadedArtifactAttached>
> + <transformers>
> + <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
> + </transformers>
> + </configuration>
> + </execution>
> + </executions>
> + </plugin>
> + </plugins>
> + </build>
> + <dependencies>
> + <dependency>
> + <groupId>${project.groupId}</groupId>
> + <artifactId>jclouds-allblobstore</artifactId>
What is the binary size overhead BlobStoreBench?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239/files#r8445725
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Shri Javadekar <no...@github.com>.
> +# Licensed to the Apache Software Foundation (ASF) under one or more
> +# contributor license agreements. See the NOTICE file distributed with
> +# this work for additional information regarding copyright ownership.
> +# The ASF licenses this file to You under the Apache License, Version 2.0
> +# (the "License"); you may not use this file except in compliance with
> +# the License. You may obtain a copy of the License at
> +#
> +# http://www.apache.org/licenses/LICENSE-2.0
> +#
> +# Unless required by applicable law or agreed to in writing, software
> +# distributed under the License is distributed on an "AS IS" BASIS,
> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> +# See the License for the specific language governing permissions and
> +# limitations under the License.
> +
> +#!/usr/bin/env bash
Done.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239/files#r8645491
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #958](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/958/) FAILURE
Looks like there's a problem with this pull request
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-31555442
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Zack Shoylev <no...@github.com>.
@jdaggett suggested we can merge some of the other repos into a tools repo. Thoughts?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30870938
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #495](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/495/) FAILURE
Looks like there's a problem with this pull request
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-31557191
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Gaul <no...@github.com>.
I believe BlobStoreBench should live in the main repository in blobstore-tools as this commit does. We could recast this in karaf/cli although that has some drawbacks, including being unfriendly for developers to work with and more importantly poor user experience. This pull request raises a greater question about how to package jclouds-related tools for administrative and debugging tasks. I would like to move towards a really-executable-jar using either airline or args4j and have some work-in-progress commits here:
https://github.com/maginatics/jclouds/tree/blobstore-cli
86f9310a40cc87b1de19468fc2e61fa0a4f0a0b7 has the most interesting bits, although airline was not up to the task the last time I looked due to airlift/airline#6.
I do not want the perfect to be the enemy of the good. Maginatics has used BlobStoreBench for over a year and it would be good to push this upstream.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30868180
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Shri Javadekar <no...@github.com>.
Closed #239.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#event-147810983
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Shri Javadekar <no...@github.com>.
> +# Blobstore credentials
> +IDENTITY="myidentity"
> +CREDENTIAL="secretCreds"
> +
> +# Container used for the test. For "transient" and "filesystem" providers,
> +# the container gets created at runtime. For others, it is expected to
> +# be present.
> +CONTAINER=jclouds-container
> +
> +# Start BlobStoreBench. Note that for some providers, you might have to add
> +# additional params below.
> +# For e.g.
> +# -Djclouds.keystone.credential-type=apiAccessKeyCredentials \
> +# -Djclouds.endpoint="http://some-endpoint.blobstore.com" \
> +# -Djclouds.strip-expect-header=true \
> +java -server \
Done.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239/files#r8645492
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Phillips <no...@github.com>.
> suggested we can merge some of the other repos into a tools repo. Thoughts?
I think it's a good suggestion, but I agree with @andrewgaul that we ideally don't want to wait until we've decided and what and how to do the merge before getting this functionality it. I was trying to split that into two steps:
1. Create a jclouds-tools or whatever repo and make this the first module in there (should be doable quickly)
2. Decide what and how to merge any other tools into this repo
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30872091
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Phillips <no...@github.com>.
> I do not want the perfect to be the enemy of the good.
I agree, but I think we can still put a little thought into where this and, indeed (as you say) related types of admin tools should go before we need to accuse ourselves of "striving for immediate perfection". I don't think the code needs to be any bigger or fatter for that.
I just don't think jclouds itself, which (unless I'm missing something) currently doesn't have the "blobstore-tools" we're talking about, or indeed _any_ other kind of tooling, is the right place. Indeed, Adrian made the move the _opposite_ way around (taking administrative stuff _out_ of jclouds) a while ago, and that still seems like a good choice for me.
Unless everyone else strongly feels this belongs in jclouds, or we find another "natural" home, I could imagine this being the first entry in a new "jclouds-admin" or "jclouds-tools" repo (sorry, @zack-shoylev ;-)). That will allow us to get this out there immediately, and we can still figure out whether karaf or cli should move there as a separate step.
Hopefully that way we can start with something good without worrying about being perfect straight away?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30869343
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Phillips <no...@github.com>.
Is there any reason this would need to be in the jclouds repo? There's nothing similar in there right now. Would [jclouds-examples](https://github.com/jclouds/jclouds-examples) perhaps be a more suitable home for this?
That already contains a collection of code that _uses_ jclouds, which seems to me to be the category into which this falls.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30822439
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Shri Javadekar <no...@github.com>.
> + <shadedClassifierName>jar-with-dependencies</shadedClassifierName>
> + <createDependencyReducedPom>false</createDependencyReducedPom>
> + <shadedArtifactAttached>true</shadedArtifactAttached>
> + <transformers>
> + <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
> + </transformers>
> + </configuration>
> + </execution>
> + </executions>
> + </plugin>
> + </plugins>
> + </build>
> + <dependencies>
> + <dependency>
> + <groupId>${project.groupId}</groupId>
> + <artifactId>jclouds-allblobstore</artifactId>
The jar file jclouds-tools-1.8.0-SNAPSHOT-jar-with-dependencies.jar is 11MB.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239/files#r8645488
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #718](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/718/) FAILURE
Looks like there's a problem with this pull request
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-31555267
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Gaul <no...@github.com>.
I will investigate creating a jclouds-tools repository to host this commit and future tools.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-38323342
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #719](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/719/) FAILURE
Looks like there's a problem with this pull request
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-31557057
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Andrew Gaul <no...@github.com>.
> +import org.kohsuke.args4j.Option;
> +
> +import com.google.common.base.Preconditions;
> +import com.google.common.base.Stopwatch;
> +import com.google.common.collect.Iterables;
> +import com.google.common.collect.Lists;
> +import com.google.common.io.ByteStreams;
> +import com.google.common.net.MediaType;
> +import com.google.common.util.concurrent.FutureCallback;
> +import com.google.common.util.concurrent.Futures;
> +import com.google.common.util.concurrent.ListenableFuture;
> +import com.google.common.util.concurrent.ListeningExecutorService;
> +import com.google.common.util.concurrent.MoreExecutors;
> +
> +/**
> + * Simple BlobStore benchmark.
Credit the original authors, @andrewgaul and @rdesai09?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239/files#r8445694
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by Zack Shoylev <no...@github.com>.
> Create a jclouds-tools or whatever repo and make this the first module in there (should be doable quickly)
> Decide what and how to merge any other tools into this repo
+1
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-30872278
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #959](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/959/) FAILURE
Looks like there's a problem with this pull request
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-31557169
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #494](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/494/) FAILURE
Looks like there's a problem with this pull request
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/239#issuecomment-31555471