Provider list not populated

[Shrinand Javadekar <sh...@maginatics.com>]

Hi,

At Maginatics, we have developed a tool called BlobstoreBench using
jclouds for measuring basic GET, PUT, DELETE performance of various
blobstores. I'm trying to make this tool a part of jclouds itself.
It's a single java file called BlobStoreBench.java.

In order to be able to use all blobstore providers and apis, I am
trying to make this part of the "allblobstores" module. The file path
is now:

jclouds/allblobstore/src/main/java/org/jclouds/blobstore_tools/BlobStoreBench.java.

The code compiles just fine. It runs just fine for the "aws-s3"
provider. However, for any other provider, I get the following error
(replace cloudfiles-us with any other provider) when creating the
blobstore context.

Exception in thread "main" java.util.NoSuchElementException: key
[cloudfiles-us] not in the list of providers or apis:
{providers=[aws-s3], apis=[s3]}
at org.jclouds.ContextBuilder.newBuilder(ContextBuilder.java:176)
at org.jclouds.blobstore_tools.BlobStoreBench.getBlobStoreContext(BlobStoreBench.java:251)
at org.jclouds.blobstore_tools.BlobStoreBench.<init>(BlobStoreBench.java:191)
at org.jclouds.blobstore_tools.BlobStoreBench.main(BlobStoreBench.java:589)

Any idea what needs to be done so that all providers and apis are available?

Thanks in advance.
-Shri

Re: Provider list not populated

[Shrinand Javadekar <sh...@maginatics.com>]

Sorry, this was internal to the tool. Basically, the tool provides an
option to either directly get/put blobs or create a signed url and use
that to get/pub blobs. I was using the signed version for transient
and filesystem blobstores. And these blobstores don't support that.

For more, see https://github.com/jclouds/jclouds/pull/239.

-Shri

On Tue, Dec 17, 2013 at 12:03 PM, Andrew Phillips <ap...@qrmedia.com> wrote:
>> Also, the reason I was seeing the failures against transient and
>> filesystem blobstores was that I was trying to do signed puts. I
>> changed that to do simple puts and it worked just fine.
>>
>> Some more polishing of the code and I'll send this out for review.
>
>
> Sorry, missing some context here. Is this something internal, or related to
> a specific JIRA issue or improvement..?
>
> ap

Re: Provider list not populated

[Andrew Phillips <ap...@qrmedia.com>]

> Also, the reason I was seeing the failures against transient and
> filesystem blobstores was that I was trying to do signed puts. I
> changed that to do simple puts and it worked just fine.
>
> Some more polishing of the code and I'll send this out for review.

Sorry, missing some context here. Is this something internal, or  
related to a specific JIRA issue or improvement..?

ap

Re: Provider list not populated

[Shrinand Javadekar <sh...@maginatics.com>]

Well, I got things to work with the shade plugin. Unless there is a
strong push against using it, I'd prefer to keep it as is.

Also, the reason I was seeing the failures against transient and
filesystem blobstores was that I was trying to do signed puts. I
changed that to do simple puts and it worked just fine.

Some more polishing of the code and I'll send this out for review.

-Shri

On Tue, Dec 17, 2013 at 5:58 AM, Andrew Phillips <ap...@qrmedia.com> wrote:
>> But I'm seeing some problems when running against the "transient" and
>> "filesystem" providers.
>>
>> For the "transient" blobstore, I see the following:
>> "Cannot retry after server error, command has exceeded retry limit 5:..."
>>
>> For the "filesystem" blobstore, I get an error saying that it can't
>> find the container even if I see the directory on the filesystem
>
>
> It's true that the shade plugin offers more options for überjars, but for
> the one particular piece of functionality we need (merging the
> META-INF/services files), the assembly plugin seems to have a handler that
> does the job well enough.
>
> At least, we haven't seen any problems with the überjars for
> jclouds-examples so far.
>
> Thanks for investigating. Curious to know what's going on, but would be
> surprised if this is related to the META-INF/services merging.
>
> ap

Re: Provider list not populated

[Andrew Phillips <ap...@qrmedia.com>]

> But I'm seeing some problems when running against the "transient" and
> "filesystem" providers.
>
> For the "transient" blobstore, I see the following:
> "Cannot retry after server error, command has exceeded retry limit 5:..."
>
> For the "filesystem" blobstore, I get an error saying that it can't
> find the container even if I see the directory on the filesystem

It's true that the shade plugin offers more options for überjars, but  
for the one particular piece of functionality we need (merging the  
META-INF/services files), the assembly plugin seems to have a handler  
that does the job well enough.

At least, we haven't seen any problems with the überjars for  
jclouds-examples so far.

Thanks for investigating. Curious to know what's going on, but would  
be surprised if this is related to the META-INF/services merging.

ap

Re: Provider list not populated

[Shrinand Javadekar <sh...@maginatics.com>]

As per [1], the maven assembly plugin isn't the best way to create an
uber jar. The maven shade plugin is better for these purposes. I tried
using the shade plugin and it seems to give me the result I want. The
providers and apis get populated in the META-INF/services. I even
tried running the tool against aws-s3 and cloudfiles-us. It worked
just fine.

But I'm seeing some problems when running against the "transient" and
"filesystem" providers.

For the "transient" blobstore, I see the following:
"Cannot retry after server error, command has exceeded retry limit 5:..."

For the "filesystem" blobstore, I get an error saying that it can't
find the container even if I see the directory on the filesystem

Digging deeper.
-Shri

[1] http://maven.apache.org/plugins/maven-assembly-plugin/

On Mon, Dec 16, 2013 at 6:35 PM, Andrew Phillips <ap...@qrmedia.com> wrote:
>> I had tried that earlier but that failed with the same problem. I
>> tried it again now and see that same problem as above. I wonder if
>> there's anything wrong with my pom.xml [1].
>>
>> I create the jar-with-dependencies using:
>> $ mvn clean compile assembly:single
>
>
> Ah, yes, that won't work just like that (well spotted) - some
> META-INF/services merging needs to happen to ensure all the providers are
> retained.
>
> See e.g. the blobstore-basics jar-with-deps definition [1] for an example.
>
> ap
>
> [1]
> https://github.com/jclouds/jclouds-examples/blob/master/blobstore-basics/src/main/assembly/jar-with-dependencies.xml#L31

Re: Provider list not populated

[Andrew Phillips <ap...@qrmedia.com>]

> I had tried that earlier but that failed with the same problem. I
> tried it again now and see that same problem as above. I wonder if
> there's anything wrong with my pom.xml [1].
>
> I create the jar-with-dependencies using:
> $ mvn clean compile assembly:single

Ah, yes, that won't work just like that (well spotted) - some  
META-INF/services merging needs to happen to ensure all the providers  
are retained.

See e.g. the blobstore-basics jar-with-deps definition [1] for an example.

ap

[1]  
https://github.com/jclouds/jclouds-examples/blob/master/blobstore-basics/src/main/assembly/jar-with-dependencies.xml#L31

Re: Provider list not populated

[Shrinand Javadekar <sh...@maginatics.com>]

I had tried that earlier but that failed with the same problem. I
tried it again now and see that same problem as above. I wonder if
there's anything wrong with my pom.xml [1].

I create the jar-with-dependencies using:
$ mvn clean compile assembly:single

Not sure if this is related, but when I run the command above, I see
may messages like the following:
...
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/DEPENDENCIES already added, skipping
[INFO] META-INF/NOTICE already added, skipping
[INFO] META-INF/LICENSE already added, skipping
...

-Shri

[1] http://pastie.org/8557084

On Mon, Dec 16, 2013 at 12:29 AM, Andrew Phillips <ap...@qrmedia.com> wrote:
> Hi Shri
>
>
>> In order to be able to use all blobstore providers and apis, I am
>> trying to make this part of the "allblobstores" module. The file path
>> is now:
>
>
> Independent of the issue you're seeing, would a benchmark be better off in a
> *separate* module that *depends on* allblobstore?
>
> From what I recall, allblobstore was never supposed to be "a thing" with any
> functionality of its own, it was just a convenient way of getting a lot of
> Maven dependencies in one go.
>
> I can see plenty of uses for a benchmarking tool, but I think you should be
> able to use that in your code (if you want to), or as a standalone tool,
> *whatever* your preferred set of blobstore dependencies. Put another way:
> you shouldn't have to drag allblobstore into your dependency list to use the
> bench.
>
> Back to your problem: could you try putting it in its *own* project which
> *depends on* allblobstore to see if that makes any difference to the set of
> providers that are available?
>
> ap

Re: Provider list not populated

[Andrew Phillips <ap...@qrmedia.com>]

Hi Shri

> In order to be able to use all blobstore providers and apis, I am
> trying to make this part of the "allblobstores" module. The file path
> is now:

Independent of the issue you're seeing, would a benchmark be better  
off in a *separate* module that *depends on* allblobstore?

 From what I recall, allblobstore was never supposed to be "a thing"  
with any functionality of its own, it was just a convenient way of  
getting a lot of Maven dependencies in one go.

I can see plenty of uses for a benchmarking tool, but I think you  
should be able to use that in your code (if you want to), or as a  
standalone tool, *whatever* your preferred set of blobstore  
dependencies. Put another way: you shouldn't have to drag allblobstore  
into your dependency list to use the bench.

Back to your problem: could you try putting it in its *own* project  
which *depends on* allblobstore to see if that makes any difference to  
the set of providers that are available?

ap