You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jclouds.apache.org by "Yao, Julie" <ju...@hpe.com> on 2016/01/29 05:53:44 UTC
Use JClouds to talk to non AWS cloud with S3 API
I am using jcloud 1.9.2. We have set up our own swift repository with s3 api. We can access the repository using boto python script
I am using the code in the following post
http://stackoverflow.com/questions/9389840/use-jclouds-to-talk-to-non-aws-cloud-with-s3-api
but I got error.:
public static void main(String args[]) {
String username = "test:tester";
String password = "testing";
Properties overrides = new Properties();
overrides.setProperty(PROPERTY_S3_VIRTUAL_HOST_BUCKETS, "false");
BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("http://myserver:8080")
.credentials(username, password)
.overrides(overrides)
.buildView(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
String bucket="public";
blobStore.containerExists(bucket);
System.out.println("bucket exist "+ bucket);
}
Exception in thread "main" org.jclouds.http.HttpResponseException: request: HEADhttp://myserver:8080/public HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
....
It throw exception on the line:
blobStore.containerExists(bucket);
Thanks
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by "Yao, Julie" <ju...@hpe.com>.
> Also, have you been able to try Ranjith's suggestion [2] and use the ApacheHC driver?
It is SUCCESS for real. Last time I tried and failed because I mixed with services/Walrus and S3Client.
I created a container and verified using s3curl.
Thanks a lot!
Julie
-----Original Message-----
From: Andrew Phillips [mailto:andrewp@apache.org]
Sent: Sunday, January 31, 2016 1:35 PM
To: user@jclouds.apache.org
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
Hm...not much difference there:
s3curl call that succeeds:
HEAD /public HTTP/1.1
User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Host: jag-itop-svr.dev.opsware.com:8080
Accept: */*
Date: Sun, 31 Jan 2016 17:54:30 +0000
Authorization: AWS test:tester:K4fukbjysjqzsArmq8EXHnc/tnE=
jclouds call that fails:
HEAD http://jag-itop-svr.dev.opsware.com:8080/public HTTP/1.1
Date: Sun, 31 Jan 2016 21:18:50 GMT
Authorization: AWS test:tester:jxcGT++8gYPHXO8Vpgt2FtIu9FI=
Can you try to reproduce the successful call using 'plain' cURL [1], then without each of the headers that are different ('User-Agent', 'Host' and 'Accept')? Hopefully, that will allow us to figure out which of those three headers is causing the 500.
Also, have you been able to try Ranjith's suggestion [2] and use the ApacheHC driver?
Regards
ap
[1] https://curl.haxx.se/
[2] http://markmail.org/message/7gw6wnpbaefgp64r
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by Andrew Phillips <an...@apache.org>.
Hm...not much difference there:
s3curl call that succeeds:
HEAD /public HTTP/1.1
User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Host: jag-itop-svr.dev.opsware.com:8080
Accept: */*
Date: Sun, 31 Jan 2016 17:54:30 +0000
Authorization: AWS test:tester:K4fukbjysjqzsArmq8EXHnc/tnE=
jclouds call that fails:
HEAD http://jag-itop-svr.dev.opsware.com:8080/public HTTP/1.1
Date: Sun, 31 Jan 2016 21:18:50 GMT
Authorization: AWS test:tester:jxcGT++8gYPHXO8Vpgt2FtIu9FI=
Can you try to reproduce the successful call using 'plain' cURL [1],
then without each of the headers that are different ('User-Agent',
'Host' and 'Accept')? Hopefully, that will allow us to figure out which
of those three headers is causing the 500.
Also, have you been able to try Ranjith's suggestion [2] and use the
ApacheHC driver?
Regards
ap
[1] https://curl.haxx.se/
[2] http://markmail.org/message/7gw6wnpbaefgp64r
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by "Yao, Julie" <ju...@hpe.com>.
Hi Andrew,
I attach the logs here. Console output:
Exception in thread "main" org.jclouds.http.HttpResponseException: request: HEAD http://jag-itop-svr.dev.opsware.com:8080/public HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:136)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy45.bucketExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.containerExists(S3BlobStore.java:130)
at com.itoa.server.test.RepositoryS31.main(RepositoryS31.java:41)
Thanks!
-----Original Message-----
From: Andrew Phillips [mailto:andrewp@apache.org]
Sent: Sunday, January 31, 2016 12:27 PM
To: Yao, Julie
Cc: user@jclouds.apache.org
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
Hi Julie
Thanks for the additional details! Good to see we're now trying to connect to the same URL in both cases (i.e.
http://jag-itop-svr.dev.opsware.com:8080/public).
Could you attach the wire and header logs from *this* failed jclouds call? Now that we have the details from s3curl, we should be able to see what is different about the request jclouds is sending.
Regards
ap
On 2016-01-31 14:53, Yao, Julie wrote:
> Debug the code
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService
> protected HttpResponse invoke(HttpURLConnection connection) throws
> IOException, InterruptedException {
> HttpResponse.Builder<?> builder = HttpResponse.builder();
> InputStream in = null;
> try {
> in = connection.getInputStream();
> } catch (IOException e) {
> in = connection.getErrorStream();
>
> detailmessage:
> Server returned HTTP response code: 500 for URL:
> http://jag-itop-svr.dev.opsware.com:8080/public
> I checked connection.authObj: null
> Is connection.authObj correct?
>
> Here is my code:
> String username = "test:tester";
> String password = "testing";
> Properties overrides = new Properties();
> overrides.setProperty(PROPERTY_S3_VIRTUAL_HOST_BUCKETS,
> "false");
> overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/");
>
> BlobStoreContext context = ContextBuilder.newBuilder(new
> S3ApiMetadata())
> .endpoint("http://jag-itop-svr.dev.opsware.com:8080")
> .credentials(username, password)
> .overrides(overrides)
> .modules(modules)
> .buildView(BlobStoreContext.class);
>
> BlobStore blobStore = context.getBlobStore();
> String bucket="public";
> boolean exist = blobStore.containerExists(bucket);
>
> I also have VM args:
> -Djclouds.trust-all-certs=true -Djclouds.relax-hostname=true
> -Djclouds.provider=s3
>
> Thanks!
>
> -----Original Message-----
> From: Yao, Julie
> Sent: Sunday, January 31, 2016 10:01 AM
> To: Andrew Phillips; user@jclouds.apache.org
> Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
>
>> Ah, OK. To confirm: are we calling a Walrus installation here?
> No
>
> I got s3curl.pl working. I have a bucket called 'public'. The rest api
> to see if the bucket 'public' exist:
> HEAD /public
>
> ./s3curl.pl --id 'test:tester' --key 'testing' --head -- -v http://
> jag-itop-svr.dev.opsware.com:8080/public
> * About to connect() to jag-itop-svr.dev.opsware.com port 8080
> * Trying 192.168.34.176... connected
> * Connected to jag-itop-svr.dev.opsware.com (192.168.34.176) port 8080
>> HEAD /public HTTP/1.1
>> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
>> OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
>> Host: jag-itop-svr.dev.opsware.com:8080
>> Accept: */*
>> Date: Sun, 31 Jan 2016 17:54:30 +0000
>> Authorization: AWS test:tester:K4fukbjysjqzsArmq8EXHnc/tnE=
>>
> < HTTP/1.1 200 OK
> HTTP/1.1 200 OK
> < x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
> x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
> < Content-Length: 0
> Content-Length: 0
> < x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
> x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
> < Content-Type: text/plain; charset=utf-8
> Content-Type: text/plain; charset=utf-8 < X-Trans-Id:
> tx29a5adaa1d8345c882658-0056ae4ab0
> X-Trans-Id: tx29a5adaa1d8345c882658-0056ae4ab0
> < Date: Sun, 31 Jan 2016 17:56:00 GMT
> Date: Sun, 31 Jan 2016 17:56:00 GMT
>
> I will download source code to debug.
>
> Thanks!
>
>
> -----Original Message-----
> From: Andrew Phillips [mailto:andrewp@apache.org]
> Sent: Friday, January 29, 2016 6:35 PM
> To: user@jclouds.apache.org
> Cc: Yao, Julie
> Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
>
>> I guess
>> overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
>> is not correct.
>> Without it, it will throw exception.
>
> Ah, OK. To confirm: are we calling a Walrus installation here?
>
> Also, in your boto code snippet:
>
> def connect(self):
> self.connection = boto.connect_s3 (
> aws_access_key_id = self.access_key_id,
> aws_secret_access_key = self.secret_access_key,
> port = self.port,
> host= self.host,
> is_secure=False,
> calling_format=boto.s3.connection.OrdinaryCallingFormat()
> )
>
> What are the values of "self.host" and "self.port"? I'm assuming they
> are "myserver" and "8080", or so? In that case, boto should also be
> looking for http://myserver:8080/julie, and not
> http://myserver:8080/services/Walrus/julie, which the jclouds settings
> are causing us to call.
>
> Please see if you can make a successful HEAD Bucket call using cURL or
> other web request generator:
>
> http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html
>
> It will be much easier to determine the correct settings for jclouds
> once we have an actual example of an HTTP request to your service that
> is successful.
>
> Regards
>
> ap
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by Andrew Phillips <an...@apache.org>.
Hi Julie
Thanks for the additional details! Good to see we're now trying to
connect to the same URL in both cases (i.e.
http://jag-itop-svr.dev.opsware.com:8080/public).
Could you attach the wire and header logs from *this* failed jclouds
call? Now that we have the details from s3curl, we should be able to see
what is different about the request jclouds is sending.
Regards
ap
On 2016-01-31 14:53, Yao, Julie wrote:
> Debug the code
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService
> protected HttpResponse invoke(HttpURLConnection connection) throws
> IOException, InterruptedException {
> HttpResponse.Builder<?> builder = HttpResponse.builder();
> InputStream in = null;
> try {
> in = connection.getInputStream();
> } catch (IOException e) {
> in = connection.getErrorStream();
>
> detailmessage:
> Server returned HTTP response code: 500 for URL:
> http://jag-itop-svr.dev.opsware.com:8080/public
> I checked connection.authObj: null
> Is connection.authObj correct?
>
> Here is my code:
> String username = "test:tester";
> String password = "testing";
> Properties overrides = new Properties();
> overrides.setProperty(PROPERTY_S3_VIRTUAL_HOST_BUCKETS,
> "false");
> overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/");
>
> BlobStoreContext context = ContextBuilder.newBuilder(new
> S3ApiMetadata())
> .endpoint("http://jag-itop-svr.dev.opsware.com:8080")
> .credentials(username, password)
> .overrides(overrides)
> .modules(modules)
> .buildView(BlobStoreContext.class);
>
> BlobStore blobStore = context.getBlobStore();
> String bucket="public";
> boolean exist = blobStore.containerExists(bucket);
>
> I also have VM args:
> -Djclouds.trust-all-certs=true -Djclouds.relax-hostname=true
> -Djclouds.provider=s3
>
> Thanks!
>
> -----Original Message-----
> From: Yao, Julie
> Sent: Sunday, January 31, 2016 10:01 AM
> To: Andrew Phillips; user@jclouds.apache.org
> Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
>
>> Ah, OK. To confirm: are we calling a Walrus installation here?
> No
>
> I got s3curl.pl working. I have a bucket called 'public'. The rest api
> to see if the bucket 'public' exist:
> HEAD /public
>
> ./s3curl.pl --id 'test:tester' --key 'testing' --head -- -v http://
> jag-itop-svr.dev.opsware.com:8080/public
> * About to connect() to jag-itop-svr.dev.opsware.com port 8080
> * Trying 192.168.34.176... connected
> * Connected to jag-itop-svr.dev.opsware.com (192.168.34.176) port 8080
>> HEAD /public HTTP/1.1
>> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
>> OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
>> Host: jag-itop-svr.dev.opsware.com:8080
>> Accept: */*
>> Date: Sun, 31 Jan 2016 17:54:30 +0000
>> Authorization: AWS test:tester:K4fukbjysjqzsArmq8EXHnc/tnE=
>>
> < HTTP/1.1 200 OK
> HTTP/1.1 200 OK
> < x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
> x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
> < Content-Length: 0
> Content-Length: 0
> < x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
> x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
> < Content-Type: text/plain; charset=utf-8
> Content-Type: text/plain; charset=utf-8
> < X-Trans-Id: tx29a5adaa1d8345c882658-0056ae4ab0
> X-Trans-Id: tx29a5adaa1d8345c882658-0056ae4ab0
> < Date: Sun, 31 Jan 2016 17:56:00 GMT
> Date: Sun, 31 Jan 2016 17:56:00 GMT
>
> I will download source code to debug.
>
> Thanks!
>
>
> -----Original Message-----
> From: Andrew Phillips [mailto:andrewp@apache.org]
> Sent: Friday, January 29, 2016 6:35 PM
> To: user@jclouds.apache.org
> Cc: Yao, Julie
> Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
>
>> I guess
>> overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
>> is not correct.
>> Without it, it will throw exception.
>
> Ah, OK. To confirm: are we calling a Walrus installation here?
>
> Also, in your boto code snippet:
>
> def connect(self):
> self.connection = boto.connect_s3 (
> aws_access_key_id = self.access_key_id,
> aws_secret_access_key = self.secret_access_key,
> port = self.port,
> host= self.host,
> is_secure=False,
> calling_format=boto.s3.connection.OrdinaryCallingFormat()
> )
>
> What are the values of "self.host" and "self.port"? I'm assuming they
> are "myserver" and "8080", or so? In that case, boto should also be
> looking for http://myserver:8080/julie, and not
> http://myserver:8080/services/Walrus/julie, which the jclouds settings
> are causing us to call.
>
> Please see if you can make a successful HEAD Bucket call using cURL or
> other web request generator:
>
> http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html
>
> It will be much easier to determine the correct settings for jclouds
> once we have an actual example of an HTTP request to your service that
> is successful.
>
> Regards
>
> ap
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by "Yao, Julie" <ju...@hpe.com>.
Debug the code
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService
protected HttpResponse invoke(HttpURLConnection connection) throws IOException, InterruptedException {
HttpResponse.Builder<?> builder = HttpResponse.builder();
InputStream in = null;
try {
in = connection.getInputStream();
} catch (IOException e) {
in = connection.getErrorStream();
detailmessage:
Server returned HTTP response code: 500 for URL: http://jag-itop-svr.dev.opsware.com:8080/public
I checked connection.authObj: null
Is connection.authObj correct?
Here is my code:
String username = "test:tester";
String password = "testing";
Properties overrides = new Properties();
overrides.setProperty(PROPERTY_S3_VIRTUAL_HOST_BUCKETS, "false");
overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/");
BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("http://jag-itop-svr.dev.opsware.com:8080")
.credentials(username, password)
.overrides(overrides)
.modules(modules)
.buildView(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
String bucket="public";
boolean exist = blobStore.containerExists(bucket);
I also have VM args:
-Djclouds.trust-all-certs=true -Djclouds.relax-hostname=true -Djclouds.provider=s3
Thanks!
-----Original Message-----
From: Yao, Julie
Sent: Sunday, January 31, 2016 10:01 AM
To: Andrew Phillips; user@jclouds.apache.org
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
> Ah, OK. To confirm: are we calling a Walrus installation here?
No
I got s3curl.pl working. I have a bucket called 'public'. The rest api to see if the bucket 'public' exist:
HEAD /public
./s3curl.pl --id 'test:tester' --key 'testing' --head -- -v http:// jag-itop-svr.dev.opsware.com:8080/public
* About to connect() to jag-itop-svr.dev.opsware.com port 8080
* Trying 192.168.34.176... connected
* Connected to jag-itop-svr.dev.opsware.com (192.168.34.176) port 8080
> HEAD /public HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
> OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: jag-itop-svr.dev.opsware.com:8080
> Accept: */*
> Date: Sun, 31 Jan 2016 17:54:30 +0000
> Authorization: AWS test:tester:K4fukbjysjqzsArmq8EXHnc/tnE=
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
< Content-Length: 0
Content-Length: 0
< x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< X-Trans-Id: tx29a5adaa1d8345c882658-0056ae4ab0
X-Trans-Id: tx29a5adaa1d8345c882658-0056ae4ab0
< Date: Sun, 31 Jan 2016 17:56:00 GMT
Date: Sun, 31 Jan 2016 17:56:00 GMT
I will download source code to debug.
Thanks!
-----Original Message-----
From: Andrew Phillips [mailto:andrewp@apache.org]
Sent: Friday, January 29, 2016 6:35 PM
To: user@jclouds.apache.org
Cc: Yao, Julie
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
> I guess
> overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
> is not correct.
> Without it, it will throw exception.
Ah, OK. To confirm: are we calling a Walrus installation here?
Also, in your boto code snippet:
def connect(self):
self.connection = boto.connect_s3 (
aws_access_key_id = self.access_key_id,
aws_secret_access_key = self.secret_access_key,
port = self.port,
host= self.host,
is_secure=False,
calling_format=boto.s3.connection.OrdinaryCallingFormat()
)
What are the values of "self.host" and "self.port"? I'm assuming they are "myserver" and "8080", or so? In that case, boto should also be looking for http://myserver:8080/julie, and not http://myserver:8080/services/Walrus/julie, which the jclouds settings are causing us to call.
Please see if you can make a successful HEAD Bucket call using cURL or other web request generator:
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html
It will be much easier to determine the correct settings for jclouds once we have an actual example of an HTTP request to your service that is successful.
Regards
ap
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by "Yao, Julie" <ju...@hpe.com>.
> Ah, OK. To confirm: are we calling a Walrus installation here?
No
I got s3curl.pl working. I have a bucket called 'public'. The rest api to see if the bucket 'public' exist:
HEAD /public
./s3curl.pl --id 'test:tester' --key 'testing' --head -- -v http:// jag-itop-svr.dev.opsware.com:8080/public
* About to connect() to jag-itop-svr.dev.opsware.com port 8080
* Trying 192.168.34.176... connected
* Connected to jag-itop-svr.dev.opsware.com (192.168.34.176) port 8080
> HEAD /public HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: jag-itop-svr.dev.opsware.com:8080
> Accept: */*
> Date: Sun, 31 Jan 2016 17:54:30 +0000
> Authorization: AWS test:tester:K4fukbjysjqzsArmq8EXHnc/tnE=
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
< Content-Length: 0
Content-Length: 0
< x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< X-Trans-Id: tx29a5adaa1d8345c882658-0056ae4ab0
X-Trans-Id: tx29a5adaa1d8345c882658-0056ae4ab0
< Date: Sun, 31 Jan 2016 17:56:00 GMT
Date: Sun, 31 Jan 2016 17:56:00 GMT
I will download source code to debug.
Thanks!
-----Original Message-----
From: Andrew Phillips [mailto:andrewp@apache.org]
Sent: Friday, January 29, 2016 6:35 PM
To: user@jclouds.apache.org
Cc: Yao, Julie
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
> I guess
> overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
> is not correct.
> Without it, it will throw exception.
Ah, OK. To confirm: are we calling a Walrus installation here?
Also, in your boto code snippet:
def connect(self):
self.connection = boto.connect_s3 (
aws_access_key_id = self.access_key_id,
aws_secret_access_key = self.secret_access_key,
port = self.port,
host= self.host,
is_secure=False,
calling_format=boto.s3.connection.OrdinaryCallingFormat()
)
What are the values of "self.host" and "self.port"? I'm assuming they are "myserver" and "8080", or so? In that case, boto should also be looking for http://myserver:8080/julie, and not http://myserver:8080/services/Walrus/julie, which the jclouds settings are causing us to call.
Please see if you can make a successful HEAD Bucket call using cURL or other web request generator:
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html
It will be much easier to determine the correct settings for jclouds once we have an actual example of an HTTP request to your service that is successful.
Regards
ap
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by Andrew Phillips <an...@apache.org>.
> I guess
> overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
> is not correct.
> Without it, it will throw exception.
Ah, OK. To confirm: are we calling a Walrus installation here?
Also, in your boto code snippet:
def connect(self):
self.connection = boto.connect_s3 (
aws_access_key_id = self.access_key_id,
aws_secret_access_key = self.secret_access_key,
port = self.port,
host= self.host,
is_secure=False,
calling_format=boto.s3.connection.OrdinaryCallingFormat()
)
What are the values of "self.host" and "self.port"? I'm assuming they
are "myserver" and "8080", or so? In that case, boto should also be
looking for http://myserver:8080/julie, and not
http://myserver:8080/services/Walrus/julie, which the jclouds settings
are causing us to call.
Please see if you can make a successful HEAD Bucket call using cURL or
other web request generator:
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html
It will be much easier to determine the correct settings for jclouds
once we have an actual example of an HTTP request to your service that
is successful.
Regards
ap
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by Andrew Phillips <an...@apache.org>.
> I attach log here.
Looking at the logs, I don't think the
overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
setting is correct :-( The logs show that jclouds is trying to call
2016-01-29 16:27:24,495 DEBUG [jclouds.headers] [main] >> PUT
http://jag-itop-svr.dev.opsware.com:8080/services/Walrus/julie/ HTTP/1.1
and the error is
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not
exist.</Message><RequestId>txf9b97455b62b4657a59fe-0056ac03af</RequestId><BucketName>services</BucketName></Error>
i.e. it's checking for a bucket named "services", rather than "julie".
Can you set up logging in boto [1] to see what the successful call is
that boto is making? Is it a HEAD to
http://jag-itop-svr.dev.opsware.com:8080/julie, or some other call?
Regards
ap
[1]
http://boto.cloudhackers.com/en/latest/getting_started.html#troubleshooting-connections
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by Andrew Phillips <an...@apache.org>.
> I tried S3Client and the result is still wrong.
Is that the right code snippet? The example you included still uses the
portable BlobStore interface?
Regards
ap
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by "Yao, Julie" <ju...@hpe.com>.
I tried S3Client and the result is still wrong.
Code:
BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("http://myserver:8080/services/Walrus")
.credentials(username, password)
.overrides(overrides)
.modules(modules)
.buildView(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
String bucket="public";
boolean exist = blobStore.containerExists(bucket);
System.out.println("bucket "+ bucket + " exist "+exist);
bucket="julie";
blobStore.createContainerInLocation(null, bucket);
exist = blobStore.containerExists(bucket);
I attach log here.
Thanks!
-----Original Message-----
From: Andrew Phillips [mailto:andrewp@apache.org]
Sent: Friday, January 29, 2016 3:23 PM
To: user@jclouds.apache.org
Cc: Yao, Julie
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
> but the result is wrong.
Ah, pity :-( Caught celebrating too soon ;-) One of the things that may be worth trying here is to move away from the portably blobstore abstraction and try the API directly - that's more comparable to the boto calls.
...
S3Client client = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("http://myserver:8080/services/Walrus")
.credentials(username, password)
.overrides(overrides)
.buildApi(S3Client.class);
client.putObject(...)
See
https://jclouds.apache.org/reference/javadoc/1.9.x/index.html?org/jclouds/s3/S3Client.html.
If that works, then we can hopefully more easily get back to the blobstore abstraction to see why that is not working.
Also, please record the wire and header logs so we can see what HTTP calls exactly are being made [1].
Regards
ap
[1] https://jclouds.apache.org/reference/logging/
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by Andrew Phillips <an...@apache.org>.
> but the result is wrong.
Ah, pity :-( Caught celebrating too soon ;-) One of the things that may
be worth trying here is to move away from the portably blobstore
abstraction and try the API directly - that's more comparable to the
boto calls.
...
S3Client client = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("http://myserver:8080/services/Walrus")
.credentials(username, password)
.overrides(overrides)
.buildApi(S3Client.class);
client.putObject(...)
See
https://jclouds.apache.org/reference/javadoc/1.9.x/index.html?org/jclouds/s3/S3Client.html.
If that works, then we can hopefully more easily get back to the
blobstore abstraction to see why that is not working.
Also, please record the wire and header logs so we can see what HTTP
calls exactly are being made [1].
Regards
ap
[1] https://jclouds.apache.org/reference/logging/
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by "Yao, Julie" <ju...@hpe.com>.
Sorry, I said "it works" too soon.
It did not throw exception when calling
blobStore.containerExists(bucket);
but the result is wrong. Bucket 'public' does exist.
When calling to create container, it throws exception.
I switched to jclouds 1.9.1. the result is the same.
Code:
public static void main(String args[]) {
String username = "test:tester";
String password = "testing";
Properties overrides = new Properties();
overrides.setProperty(PROPERTY_S3_VIRTUAL_HOST_BUCKETS, "false");
overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("http://myserver:8080/services/Walrus")
.credentials(username, password)
.overrides(overrides)
.buildView(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
String bucket="public";
boolean exist = blobStore.containerExists(bucket);
System.out.println("bucket "+ bucket + " exist "+exist);
bucket="julie";
blobStore.createContainerInLocation(null, bucket); // line 41 exception is threw
exist = blobStore.containerExists(bucket);
System.out.println("bucket exist "+ bucket + " "+exist);
}
Output:
bucket public exist false
Exception in thread "main" org.jclouds.blobstore.ContainerNotFoundException: julie not found: The specified bucket does not exist.
at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:89)
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:136)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy45.putBucketInRegion(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.createContainerInLocation(S3BlobStore.java:375)
at org.jclouds.s3.blobstore.S3BlobStore.createContainerInLocation(S3BlobStore.java:143)
at com.itoa.server.test.RepositoryS3.main(RepositoryS3.java:41)
Thanks!
-----Original Message-----
From: Yao, Julie
Sent: Friday, January 29, 2016 11:53 AM
To: user@jclouds.apache.org
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
It works now by adding following code:
overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus"); BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("myserver:8080/services/Walrus")
....
Thanks!
-----Original Message-----
From: Yao, Julie
Sent: Friday, January 29, 2016 10:49 AM
To: user@jclouds.apache.org
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
Code snippet using boto:
def connect(self):
self.connection = boto.connect_s3 (
aws_access_key_id = self.access_key_id,
aws_secret_access_key = self.secret_access_key,
port = self.port,
host= self.host,
is_secure=False,
calling_format=boto.s3.connection.OrdinaryCallingFormat()
)
bucket = self.connection.get_bucket(self.container)
....
Exception stack using jclouds:
Exception in thread "main" org.jclouds.http.HttpResponseException: request: HEAD http://myserver:8080/public HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:136)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy45.bucketExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.containerExists(S3BlobStore.java:130)
at com.itoa.server.test.RepositoryS3.main(RepositoryS3.java:32)
Thanks!
-----Original Message-----
From: Andrew Phillips [mailto:andrewp@apache.org]
Sent: Thursday, January 28, 2016 9:05 PM
To: user@jclouds.apache.org
Cc: Yao, Julie
Subject: Re: Use JClouds to talk to non AWS cloud with S3 API
Hi Julie
Could you enable wire and debug logging so we can see exactly which calls jclouds is making [1]? Also, could you share a code snippet of how you are successfully using boto to do the same thing? Hopefully, we can find out where the calls from jclouds differ from what boto is doing...
Regards
ap
[1] https://jclouds.apache.org/reference/logging/
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by Andrew Phillips <an...@apache.org>.
Thanks for the heads-up, Julie - glad you were able to get it to work!
ap
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by "Yao, Julie" <ju...@hpe.com>.
It works now by adding following code:
overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("myserver:8080/services/Walrus")
....
Thanks!
-----Original Message-----
From: Yao, Julie
Sent: Friday, January 29, 2016 10:49 AM
To: user@jclouds.apache.org
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API
Code snippet using boto:
def connect(self):
self.connection = boto.connect_s3 (
aws_access_key_id = self.access_key_id,
aws_secret_access_key = self.secret_access_key,
port = self.port,
host= self.host,
is_secure=False,
calling_format=boto.s3.connection.OrdinaryCallingFormat()
)
bucket = self.connection.get_bucket(self.container)
....
Exception stack using jclouds:
Exception in thread "main" org.jclouds.http.HttpResponseException: request: HEAD http://myserver:8080/public HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:136)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy45.bucketExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.containerExists(S3BlobStore.java:130)
at com.itoa.server.test.RepositoryS3.main(RepositoryS3.java:32)
Thanks!
-----Original Message-----
From: Andrew Phillips [mailto:andrewp@apache.org]
Sent: Thursday, January 28, 2016 9:05 PM
To: user@jclouds.apache.org
Cc: Yao, Julie
Subject: Re: Use JClouds to talk to non AWS cloud with S3 API
Hi Julie
Could you enable wire and debug logging so we can see exactly which calls jclouds is making [1]? Also, could you share a code snippet of how you are successfully using boto to do the same thing? Hopefully, we can find out where the calls from jclouds differ from what boto is doing...
Regards
ap
[1] https://jclouds.apache.org/reference/logging/
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by "Yao, Julie" <ju...@hpe.com>.
Code snippet using boto:
def connect(self):
self.connection = boto.connect_s3 (
aws_access_key_id = self.access_key_id,
aws_secret_access_key = self.secret_access_key,
port = self.port,
host= self.host,
is_secure=False,
calling_format=boto.s3.connection.OrdinaryCallingFormat()
)
bucket = self.connection.get_bucket(self.container)
....
Exception stack using jclouds:
Exception in thread "main" org.jclouds.http.HttpResponseException: request: HEAD http://myserver:8080/public HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:136)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy45.bucketExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.containerExists(S3BlobStore.java:130)
at com.itoa.server.test.RepositoryS3.main(RepositoryS3.java:32)
Thanks!
-----Original Message-----
From: Andrew Phillips [mailto:andrewp@apache.org]
Sent: Thursday, January 28, 2016 9:05 PM
To: user@jclouds.apache.org
Cc: Yao, Julie
Subject: Re: Use JClouds to talk to non AWS cloud with S3 API
Hi Julie
Could you enable wire and debug logging so we can see exactly which calls jclouds is making [1]? Also, could you share a code snippet of how you are successfully using boto to do the same thing? Hopefully, we can find out where the calls from jclouds differ from what boto is doing...
Regards
ap
[1] https://jclouds.apache.org/reference/logging/
RE: Use JClouds to talk to non AWS cloud with S3 API
Posted by "Yao, Julie" <ju...@hpe.com>.
Hi Ranjith,
It worked like magic. Three days ago, I tried it mixed with services/Walrus and S3Client and it failed. Today I cleaned up code and it worked.
Thanks!
Julie
From: Ranjith R [mailto:ranjitr1@gmail.com]
Sent: Thursday, January 28, 2016 9:57 PM
To: user@jclouds.apache.org
Cc: Yao, Julie
Subject: Re: Use JClouds to talk to non AWS cloud with S3 API
We did face a similar issue in the past, and found that the accept header set by the HttpClient was being rejected by the Swift3 middleware. We replaced it with apache httpclient, and it worked. Perhaps you could give it a try.
Iterable<Module> modules = ImmutableSet.<Module> of(new ApacheHCHttpCommandExecutorServiceModule());
BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("http://myserver:8080<http://myserver:8080/>")
.credentials(username, password)
.overrides(overrides)
.modules(modules)
.buildView(BlobStoreContext.class);
Regards,
Ranjith
On Fri, Jan 29, 2016 at 10:34 AM, Andrew Phillips <an...@apache.org>> wrote:
Hi Julie
Could you enable wire and debug logging so we can see exactly which calls jclouds is making [1]? Also, could you share a code snippet of how you are successfully using boto to do the same thing? Hopefully, we can find out where the calls from jclouds differ from what boto is doing...
Regards
ap
[1] https://jclouds.apache.org/reference/logging/
Re: Use JClouds to talk to non AWS cloud with S3 API
Posted by Ranjith R <ra...@gmail.com>.
We did face a similar issue in the past, and found that the accept header
set by the HttpClient was being rejected by the Swift3 middleware. We
replaced it with apache httpclient, and it worked. Perhaps you could give
it a try.
Iterable<Module> modules = ImmutableSet.<Module> of(new
ApacheHCHttpCommandExecutorServiceModule());
BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("http://myserver:8080")
.credentials(username, password)
.overrides(overrides)
.modules(modules)
.buildView(BlobStoreContext.class);
Regards,
Ranjith
On Fri, Jan 29, 2016 at 10:34 AM, Andrew Phillips <an...@apache.org>
wrote:
> Hi Julie
>
> Could you enable wire and debug logging so we can see exactly which calls
> jclouds is making [1]? Also, could you share a code snippet of how you are
> successfully using boto to do the same thing? Hopefully, we can find out
> where the calls from jclouds differ from what boto is doing...
>
> Regards
>
> ap
>
> [1] https://jclouds.apache.org/reference/logging/
>
Re: Use JClouds to talk to non AWS cloud with S3 API
Posted by Andrew Phillips <an...@apache.org>.
Hi Julie
Could you enable wire and debug logging so we can see exactly which
calls jclouds is making [1]? Also, could you share a code snippet of how
you are successfully using boto to do the same thing? Hopefully, we can
find out where the calls from jclouds differ from what boto is doing...
Regards
ap
[1] https://jclouds.apache.org/reference/logging/