You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@taverna.apache.org by Nadeesh Dilanga <na...@gmail.com> on 2016/06/04 04:10:17 UTC
Finalize Docker Invoke JSON format
Hi all,
I am starting this thread to discuss and finalize the docker commands we
need to expose for client side(Taverna).
Latest stable docker remote API is version 1.23[1]. And it has several APIs
that can be useful.
The original JIRA [2] mentioned about the JSON format to a docker run. I
hope it meant about the docker config.json ?
Because, given we use remote APIs, I would like to know what are the
expectations are ?
1. Do we assume that Images are created and published to the registry.
2. Do we assume that docker container is created
Given #1 and #2 done, then we are talking about starting the
container(~docker run). If that is the case, when we use remote APIs we
only need following, and no need of a JSON:
Request: POST /containers/(id or name)/start
Response: HTTP/1.1 204 No Content
There are other responses too:
Status Codes:
- *204* – no error
- *304* – container already started
- *404* – no such container
- *500* – server error
[1] - https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 04-Jun-16 11:39, Stian Soiland-Reyes wrote:
> Hi!
>
[snip]
> What is the Docker API for setting up a container? Do you need to pull the
> images first?
That is in the REST API referenced by Nadeesh -
https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/
> It would be cool if we can pull at the beginning of a
> workflow run (e.g. asynchronously when an Activity is configured) so that
> does not need to delay execution at a later step.
It may be worth checking with the Insigneo people from Sheffield who do
something analogous for VMs created to run services for workflows.
> We will probably need a volume mount of a temporary directory, so that
> input and output files can be provided to the command, but if you prefer it
> might be easier to start with stdin and stdout support; similar to the Tool
> Activity.
Alan
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi Alan,
Thanks for the feedback.
I have sent a pull request (created from a separate branch) which has the
docker test fixes. With this the unit tests works out of the box. Right now
docker activity test has inspect image.(implementation support many
including create/start etc). I will be adding tests to cover each one by
one soon.
On Fri, Jun 24, 2016 at 7:01 AM, Alan Williams <al...@googlemail.com>
wrote:
> On 24-Jun-16 06:58, Nadeesh Dilanga wrote:
>
>> Hi all,
>>
>
> Hello
>
> Yes, SSL issue also went away. When I point cert path in docker-java it
>> picks it up without any concern.
>>
>
> Great.
>
> At the moment the cert-path in TestDockerCommands is hard-coded with
> values like "/Users/Nadeesh/Documents/docker/" . In order for other people
> to run it, you need to change these.
>
> I sent a pull request with latest code changes. And now the
>> RemoteClient(Utility client for docker commands) supports
>> - Inspect an Image
>> - Search Images
>> - List all Containers
>> - Create Container
>> - Start Container
>> - Stop Container
>>
>>
>> Pulling an Image is also supported, not implemented yet. Will do that
>> also.
>> And obviously we can keep adding more capability. And in the activity
>> plugin execute, I am expecting an input "action" and based on it's value I
>> am executing the docker commands and putting a JSON String as output. For
>> errors, It is a error message.
>>
>
> OK. We can look at the execution once we are able to do the container and
> image :)
>
> Please let me know your feedback on how I can further improve the code
>> base/structure. I will be keep adding remaining docker command support for
>> RemoteClient as well as Activity.
>>
>
> Good work.
>
> Alan
>
>
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 24-Jun-16 06:58, Nadeesh Dilanga wrote:
> Hi all,
Hello
> Yes, SSL issue also went away. When I point cert path in docker-java it
> picks it up without any concern.
Great.
At the moment the cert-path in TestDockerCommands is hard-coded with
values like "/Users/Nadeesh/Documents/docker/" . In order for other
people to run it, you need to change these.
> I sent a pull request with latest code changes. And now the
> RemoteClient(Utility client for docker commands) supports
> - Inspect an Image
> - Search Images
> - List all Containers
> - Create Container
> - Start Container
> - Stop Container
>
>
> Pulling an Image is also supported, not implemented yet. Will do that also.
> And obviously we can keep adding more capability. And in the activity
> plugin execute, I am expecting an input "action" and based on it's value I
> am executing the docker commands and putting a JSON String as output. For
> errors, It is a error message.
OK. We can look at the execution once we are able to do the container
and image :)
> Please let me know your feedback on how I can further improve the code
> base/structure. I will be keep adding remaining docker command support for
> RemoteClient as well as Activity.
Good work.
Alan
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi all,
Yes, SSL issue also went away. When I point cert path in docker-java it
picks it up without any concern.
I sent a pull request with latest code changes. And now the
RemoteClient(Utility client for docker commands) supports
- Inspect an Image
- Search Images
- List all Containers
- Create Container
- Start Container
- Stop Container
Pulling an Image is also supported, not implemented yet. Will do that also.
And obviously we can keep adding more capability. And in the activity
plugin execute, I am expecting an input "action" and based on it's value I
am executing the docker commands and putting a JSON String as output. For
errors, It is a error message.
Please let me know your feedback on how I can further improve the code
base/structure. I will be keep adding remaining docker command support for
RemoteClient as well as Activity.
On Thu, Jun 23, 2016 at 6:22 AM, Stian Soiland-Reyes <st...@apache.org>
wrote:
> On 23 June 2016 at 07:17, Nadeesh Dilanga <na...@gmail.com> wrote:
> > With the help of maven dependency tree, was able to rule out all possible
> > places that packs in old jacksons. And since you mentioned about 2.7.4, I
> > only added following and was able to get rid of above error. So that
> means
> > docker-java can work with 2.7.4. I am in the process of doing code
> changes
> > accordingly to implement docker-java utility instead of docker-http.
>
> Great investigation!
>
> Dependency version management can be a bit tricky, but luckily most
> Java libraries are (somewhat) following Semantic Versioning
> http://semver.org/ which makes it a lot easier.
>
> There would be ways around it if we really needed to - but that would
> be more of a hack (e.g. shading/bundling Jackson 3.0.0 class files
> inside our own JAR but with new package names)
>
>
> > Now I am creating a full pledged DockerContainerConfiguration bean with
> > all supporting params. Will send a pull request as soon as I completes
> it.
>
> Great news! So if you find the docker-java working well then we don't
> need to focus so much on the Docker API (which I guess could change
> over time as Docker folks move rapidly)
>
> What about the SSL certificate issue, that is gone now?
>
> > One question, docker container config, that we expect to read as a JSON,
> > but we need certain params to setup docker client to invoke docker remote
> > api. Ex: remote host;port, cert path, if we have a special registry, its
> > url, username/password etc. Shall I accept these as Activity Config
> > params(the map I get in executeAsync()) ? Or any suggestions for that ?
>
> Perhaps the best would be if we have a way to override.. something like:
>
>
> 1) Is it provided as a System.getProperty() ? (e.g. -D
> taverna.docker.host=asdkfalsfkd ) -- this could then be used with
> executeworkflow.sh
>
> if not:
>
> 2) Does the activity hard-code it in its config JSON? (e.g. {
> "docker": { "host": "asdfgb" } } ))
>
> if not:
>
> 3) Is there a workbench-wide Configuration (e.g. loaded from
> ~/.taverna/conf/docker.conf) (You've already added support for this)
>
> or perhaps finally:
>
> 4) A reasonable default for this operating system?
>
> or at least:
>
> 5) Fall over very hard
>
>
> Perhaps if you put some kind of "configuration joiner" method
> somewhere, then that could also be inspected in the workbench to see
> what the "final" configuration is for an inserted Docker activity.
>
>
> also see what the docker-java already provides for such configuration
> settings/files (if any) -- perhaps we could use that directly
> instead.
>
> --
> Stian Soiland-Reyes
> Apache Taverna (incubating), Apache Commons
> http://orcid.org/0000-0001-9842-9718
>
Re: Finalize Docker Invoke JSON format
Posted by Stian Soiland-Reyes <st...@apache.org>.
On 23 June 2016 at 07:17, Nadeesh Dilanga <na...@gmail.com> wrote:
> With the help of maven dependency tree, was able to rule out all possible
> places that packs in old jacksons. And since you mentioned about 2.7.4, I
> only added following and was able to get rid of above error. So that means
> docker-java can work with 2.7.4. I am in the process of doing code changes
> accordingly to implement docker-java utility instead of docker-http.
Great investigation!
Dependency version management can be a bit tricky, but luckily most
Java libraries are (somewhat) following Semantic Versioning
http://semver.org/ which makes it a lot easier.
There would be ways around it if we really needed to - but that would
be more of a hack (e.g. shading/bundling Jackson 3.0.0 class files
inside our own JAR but with new package names)
> Now I am creating a full pledged DockerContainerConfiguration bean with
> all supporting params. Will send a pull request as soon as I completes it.
Great news! So if you find the docker-java working well then we don't
need to focus so much on the Docker API (which I guess could change
over time as Docker folks move rapidly)
What about the SSL certificate issue, that is gone now?
> One question, docker container config, that we expect to read as a JSON,
> but we need certain params to setup docker client to invoke docker remote
> api. Ex: remote host;port, cert path, if we have a special registry, its
> url, username/password etc. Shall I accept these as Activity Config
> params(the map I get in executeAsync()) ? Or any suggestions for that ?
Perhaps the best would be if we have a way to override.. something like:
1) Is it provided as a System.getProperty() ? (e.g. -D
taverna.docker.host=asdkfalsfkd ) -- this could then be used with
executeworkflow.sh
if not:
2) Does the activity hard-code it in its config JSON? (e.g. {
"docker": { "host": "asdfgb" } } ))
if not:
3) Is there a workbench-wide Configuration (e.g. loaded from
~/.taverna/conf/docker.conf) (You've already added support for this)
or perhaps finally:
4) A reasonable default for this operating system?
or at least:
5) Fall over very hard
Perhaps if you put some kind of "configuration joiner" method
somewhere, then that could also be inspected in the workbench to see
what the "final" configuration is for an inserted Docker activity.
also see what the docker-java already provides for such configuration
settings/files (if any) -- perhaps we could use that directly
instead.
--
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons
http://orcid.org/0000-0001-9842-9718
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi,
With the help of maven dependency tree, was able to rule out all possible
places that packs in old jacksons. And since you mentioned about 2.7.4, I
only added following and was able to get rid of above error. So that means
docker-java can work with 2.7.4. I am in the process of doing code changes
accordingly to implement docker-java utility instead of docker-http.
And I was able to complete the implementation for basic Docker
1. Inspect an Image
2. Create a container from a given image.
And was able to successfully execute these from docker-jave utility I
implemented. I verified the api invocation outputs with raw docker commands
and it was a success.
Now I am creating a full pledged DockerContainerConfiguration bean with
all supporting params. Will send a pull request as soon as I completes it.
One question, docker container config, that we expect to read as a JSON,
but we need certain params to setup docker client to invoke docker remote
api. Ex: remote host;port, cert path, if we have a special registry, its
url, username/password etc. Shall I accept these as Activity Config
params(the map I get in executeAsync()) ? Or any suggestions for that ?
<dependency>
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>jackson-jaxrs-json-provider</artifactId>
<version>2.7.4</version>
</dependency>
On Wed, Jun 22, 2016 at 1:06 PM, Stian Soiland-Reyes <st...@apache.org>
wrote:
> It might be tricky to upgrade to Jackson 3 as we use Jackson 2.7.4 in
> Taverna Language (the Activity's Configuration) - and so this is sadly
> part of our API and we would need to see what could break moving to
> Jackson 3. What is their breaking changes since they are bumping the
> major version?
>
> (Perhaps we should instead use a neutral JSON API like
> javax.json.JsonObject ?
> http://docs.oracle.com/javaee/7/api/javax/json/JsonObject.html --
> Apache Johnzon is one implementation)
>
>
>
> On 22 June 2016 at 05:57, Nadeesh Dilanga <na...@gmail.com> wrote:
> > Hi Alan,
> > Thank you for the reply. I already started putting together docker-java.
> > Seems there is a jackson dependency issue in latest 3.0.0 version. Trying
> > to figure out how to fix this. Will update as soon as I get this sorted
> out.
> >
> > Thank you very much for the advice on change of path. Actually Java API
> > should be straight forward.
> >
> > See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> > Exception in thread "main" java.lang.NoSuchMethodError:
> >
> com.fasterxml.jackson.databind.ObjectReader.forType(Lcom/fasterxml/jackson/databind/JavaType;)Lcom/fasterxml/jackson/databind/ObjectReader;
> > at
> >
> com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:799)
> > at
> >
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:264)
> > at
> >
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:234)
> > at
> >
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:154)
> > at
> >
> org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1124)
> > at
> >
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:851)
> > at
> >
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:783)
> > at
> >
> org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:326)
> > at
> >
> org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:786)
> > at
> >
> org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:91)
> > at
> >
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:683)
> > at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> > at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> > at org.glassfish.
> >
> >
> >
> >
> >
> > On Tue, Jun 21, 2016 at 6:08 AM, Alan Williams <alaninmcr@googlemail.com
> >
> > wrote:
> >
> >> On 18-Jun-16 08:12, Nadeesh Dilanga wrote:
> >>
> >>> Hi Alan, Hi Stian,
> >>> Please refer my latest commit @
> >>>
> >>>
> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker/taverna-docker-activity
> >>>
> >>> where I have implemented reading a injected configuration. Can you
> please
> >>> review this and let me know what I am missing here. But one thing I
> would
> >>> like to know is, who is responsible of creating(populating) the
> >>> DockerContainerConfiguration ?
> >>>
> >>
> >> In the workbench, users are able to change such configurations. That is
> >> normally done under the File -> Preferences. I think for GSOC it is
> enough
> >> to specify what needs to be in the configuration.
> >>
> >> We have to allow user to give a docker.conf
> >>> and from which some one construct the DockerContainerConfiguration and
> >>> inject it to the activity plugin.
> >>>
> >>
> >> Yes, long term we do. However, for GSOC it is enough to take a
> >> hand-written docker.conf
> >>
> >> Your DockerContainerConfigurationImpl certainly looks to have the
> correct
> >> type of information.
> >>
> >> Then I went through the taverna-engine repo code base looking for the
> clue
> >>> Stian gave, where I have to implement Configurable interface, and use
> >>> ConfigurationManager. And Configuration manager interface had
> >>> store/populate methods to override, but I found it bit unclear to
> figure
> >>> out how exactly I can use that to my use case/how it works/relation
> ship
> >>> between Configurable interface and ConfigurationManager. Do we have any
> >>> documentation on that ?
> >>>
> >>
> >> I am not sure. Stian and Gale may know.
> >>
> >> For SSL issue, I am calling the container as
> >>> https://192.168.99.100:2376/containers/create where 192.168.99.100
> is my
> >>> container host. I assume that is the target you meant ?
> >>>
> >>
> >> Yes.
> >>
> >> Alan
> >>
> >>
>
>
>
> --
> Stian Soiland-Reyes
> Apache Taverna (incubating), Apache Commons
> http://orcid.org/0000-0001-9842-9718
>
Re: Finalize Docker Invoke JSON format
Posted by Stian Soiland-Reyes <st...@apache.org>.
It might be tricky to upgrade to Jackson 3 as we use Jackson 2.7.4 in
Taverna Language (the Activity's Configuration) - and so this is sadly
part of our API and we would need to see what could break moving to
Jackson 3. What is their breaking changes since they are bumping the
major version?
(Perhaps we should instead use a neutral JSON API like
javax.json.JsonObject ?
http://docs.oracle.com/javaee/7/api/javax/json/JsonObject.html --
Apache Johnzon is one implementation)
On 22 June 2016 at 05:57, Nadeesh Dilanga <na...@gmail.com> wrote:
> Hi Alan,
> Thank you for the reply. I already started putting together docker-java.
> Seems there is a jackson dependency issue in latest 3.0.0 version. Trying
> to figure out how to fix this. Will update as soon as I get this sorted out.
>
> Thank you very much for the advice on change of path. Actually Java API
> should be straight forward.
>
> See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.fasterxml.jackson.databind.ObjectReader.forType(Lcom/fasterxml/jackson/databind/JavaType;)Lcom/fasterxml/jackson/databind/ObjectReader;
> at
> com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:799)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:264)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:234)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:154)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1124)
> at
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:851)
> at
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:783)
> at
> org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:326)
> at
> org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:786)
> at
> org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:91)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:683)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.
>
>
>
>
>
> On Tue, Jun 21, 2016 at 6:08 AM, Alan Williams <al...@googlemail.com>
> wrote:
>
>> On 18-Jun-16 08:12, Nadeesh Dilanga wrote:
>>
>>> Hi Alan, Hi Stian,
>>> Please refer my latest commit @
>>>
>>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker/taverna-docker-activity
>>>
>>> where I have implemented reading a injected configuration. Can you please
>>> review this and let me know what I am missing here. But one thing I would
>>> like to know is, who is responsible of creating(populating) the
>>> DockerContainerConfiguration ?
>>>
>>
>> In the workbench, users are able to change such configurations. That is
>> normally done under the File -> Preferences. I think for GSOC it is enough
>> to specify what needs to be in the configuration.
>>
>> We have to allow user to give a docker.conf
>>> and from which some one construct the DockerContainerConfiguration and
>>> inject it to the activity plugin.
>>>
>>
>> Yes, long term we do. However, for GSOC it is enough to take a
>> hand-written docker.conf
>>
>> Your DockerContainerConfigurationImpl certainly looks to have the correct
>> type of information.
>>
>> Then I went through the taverna-engine repo code base looking for the clue
>>> Stian gave, where I have to implement Configurable interface, and use
>>> ConfigurationManager. And Configuration manager interface had
>>> store/populate methods to override, but I found it bit unclear to figure
>>> out how exactly I can use that to my use case/how it works/relation ship
>>> between Configurable interface and ConfigurationManager. Do we have any
>>> documentation on that ?
>>>
>>
>> I am not sure. Stian and Gale may know.
>>
>> For SSL issue, I am calling the container as
>>> https://192.168.99.100:2376/containers/create where 192.168.99.100 is my
>>> container host. I assume that is the target you meant ?
>>>
>>
>> Yes.
>>
>> Alan
>>
>>
--
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons
http://orcid.org/0000-0001-9842-9718
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi Alan,
Thank you for the reply. I already started putting together docker-java.
Seems there is a jackson dependency issue in latest 3.0.0 version. Trying
to figure out how to fix this. Will update as soon as I get this sorted out.
Thank you very much for the advice on change of path. Actually Java API
should be straight forward.
See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.NoSuchMethodError:
com.fasterxml.jackson.databind.ObjectReader.forType(Lcom/fasterxml/jackson/databind/JavaType;)Lcom/fasterxml/jackson/databind/ObjectReader;
at
com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:799)
at
org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:264)
at
org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:234)
at
org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:154)
at
org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1124)
at
org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:851)
at
org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:783)
at
org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:326)
at
org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:786)
at
org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:91)
at
org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:683)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.
On Tue, Jun 21, 2016 at 6:08 AM, Alan Williams <al...@googlemail.com>
wrote:
> On 18-Jun-16 08:12, Nadeesh Dilanga wrote:
>
>> Hi Alan, Hi Stian,
>> Please refer my latest commit @
>>
>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker/taverna-docker-activity
>>
>> where I have implemented reading a injected configuration. Can you please
>> review this and let me know what I am missing here. But one thing I would
>> like to know is, who is responsible of creating(populating) the
>> DockerContainerConfiguration ?
>>
>
> In the workbench, users are able to change such configurations. That is
> normally done under the File -> Preferences. I think for GSOC it is enough
> to specify what needs to be in the configuration.
>
> We have to allow user to give a docker.conf
>> and from which some one construct the DockerContainerConfiguration and
>> inject it to the activity plugin.
>>
>
> Yes, long term we do. However, for GSOC it is enough to take a
> hand-written docker.conf
>
> Your DockerContainerConfigurationImpl certainly looks to have the correct
> type of information.
>
> Then I went through the taverna-engine repo code base looking for the clue
>> Stian gave, where I have to implement Configurable interface, and use
>> ConfigurationManager. And Configuration manager interface had
>> store/populate methods to override, but I found it bit unclear to figure
>> out how exactly I can use that to my use case/how it works/relation ship
>> between Configurable interface and ConfigurationManager. Do we have any
>> documentation on that ?
>>
>
> I am not sure. Stian and Gale may know.
>
> For SSL issue, I am calling the container as
>> https://192.168.99.100:2376/containers/create where 192.168.99.100 is my
>> container host. I assume that is the target you meant ?
>>
>
> Yes.
>
> Alan
>
>
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 18-Jun-16 08:12, Nadeesh Dilanga wrote:
> Hi Alan, Hi Stian,
> Please refer my latest commit @
> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker/taverna-docker-activity
>
> where I have implemented reading a injected configuration. Can you please
> review this and let me know what I am missing here. But one thing I would
> like to know is, who is responsible of creating(populating) the
> DockerContainerConfiguration ?
In the workbench, users are able to change such configurations. That is
normally done under the File -> Preferences. I think for GSOC it is
enough to specify what needs to be in the configuration.
> We have to allow user to give a docker.conf
> and from which some one construct the DockerContainerConfiguration and
> inject it to the activity plugin.
Yes, long term we do. However, for GSOC it is enough to take a
hand-written docker.conf
Your DockerContainerConfigurationImpl certainly looks to have the
correct type of information.
> Then I went through the taverna-engine repo code base looking for the clue
> Stian gave, where I have to implement Configurable interface, and use
> ConfigurationManager. And Configuration manager interface had
> store/populate methods to override, but I found it bit unclear to figure
> out how exactly I can use that to my use case/how it works/relation ship
> between Configurable interface and ConfigurationManager. Do we have any
> documentation on that ?
I am not sure. Stian and Gale may know.
> For SSL issue, I am calling the container as
> https://192.168.99.100:2376/containers/create where 192.168.99.100 is my
> container host. I assume that is the target you meant ?
Yes.
Alan
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 20-Jun-16 06:47, Nadeesh Dilanga wrote:
> Hi all, one more thing to add.
> Create container is the most heavy API call(payload/parameter size). But
> how about things like start/stop/search/info/delete/ like methods. Should
> be think on accepting them as as activity plugin configs ?
docker-java has these, so I think it would be best to try to use that.
What extra information is needed to start the container?
Alan
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 21-Jun-16 08:21, Nadeesh Dilanga wrote:
> Hi Stain, Hi Alan,
> Can you please review the code with my latest commit @
> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
> where I came up with a Configuration Manager and also implemented
> "Configurable".
I think you need a separate interface called
DockerContainerConfiguration as well as the class
DockerContainerConfigurationImpl. DockerActivity should then take a
DockerContainerConfiguration not a DockerContainerConfigurationImpl.
> I would like to add a unit test case for this by creating a
> DockerContainerConfigurationImpl and inject a ConfigurationManager. Can you
> please point me to such example. I tried, but could not find anything yet.
> i.e I was not able to find a usage of "DatabaseConfigurationImpl" in
> taverna-engine/taverna-osgi/taverna-common-activities repos.
DatabaseConfiguration and DatabaseConfigurationImpl are in the Taverna
engine not in the common activities. For the unit test, I think it is OK
to create an explicit DockerContainerConfigurationImpl and insert that
into the DockerActivity.
> Because I am planning to have a proper test case before the mid term
> evaluation deadline. Also appreciate if you can provide your
> feedback/expectations on this project's mid point
I will be satisfied if you can retrieve the metadata about an image and,
if possible, talk to a container.
> and if it has any
> required TODOs I can complete them ASAP before the mid term evaluation
> deadline.
Rather than implement your own REST calls to talk to a container, have
you looked at https://github.com/docker-java/docker-java ? If it is too
difficult to switch, then I think your REST utils are OK for the moment,
Alan
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi,
I also sent a pull request
https://github.com/apache/incubator-taverna-common-activities/pull/5 with
my current four commits.
On Tue, Jun 21, 2016 at 3:21 AM, Nadeesh Dilanga <na...@gmail.com>
wrote:
> Hi Stain, Hi Alan,
> Can you please review the code with my latest commit @
> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
> where I came up with a Configuration Manager and also implemented
> "Configurable".
>
> I would like to add a unit test case for this by creating a
> DockerContainerConfigurationImpl and inject a ConfigurationManager. Can you
> please point me to such example. I tried, but could not find anything yet.
> i.e I was not able to find a usage of "DatabaseConfigurationImpl" in
> taverna-engine/taverna-osgi/taverna-common-activities repos.
>
> Because I am planning to have a proper test case before the mid term
> evaluation deadline. Also appreciate if you can provide your
> feedback/expectations on this project's mid point and if it has any
> required TODOs I can complete them ASAP before the mid term evaluation
> deadline.
>
>
> On Mon, Jun 20, 2016 at 1:47 AM, Nadeesh Dilanga <na...@gmail.com>
> wrote:
>
>> Hi all, one more thing to add.
>> Create container is the most heavy API call(payload/parameter size). But
>> how about things like start/stop/search/info/delete/ like methods. Should
>> be think on accepting them as as activity plugin configs ?
>>
>>
>>
>>
>>
>>
>> On Sat, Jun 18, 2016 at 3:12 AM, Nadeesh Dilanga <na...@gmail.com>
>> wrote:
>>
>>> Hi Alan, Hi Stian,
>>> Please refer my latest commit @
>>>
>>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker/taverna-docker-activity
>>>
>>> where I have implemented reading a injected configuration. Can you
>>> please review this and let me know what I am missing here. But one thing I
>>> would like to know is, who is responsible of creating(populating) the
>>> DockerContainerConfiguration ? We have to allow user to give a docker.conf
>>> and from which some one construct the DockerContainerConfiguration and
>>> inject it to the activity plugin.
>>>
>>> Then I went through the taverna-engine repo code base looking for the
>>> clue Stian gave, where I have to implement Configurable interface, and use
>>> ConfigurationManager. And Configuration manager interface had
>>> store/populate methods to override, but I found it bit unclear to figure
>>> out how exactly I can use that to my use case/how it works/relation ship
>>> between Configurable interface and ConfigurationManager. Do we have any
>>> documentation on that ?
>>>
>>> For SSL issue, I am calling the container as
>>> https://192.168.99.100:2376/containers/create where 192.168.99.100 is
>>> my container host. I assume that is the target you meant ?
>>>
>>>
>>>
>>> On Fri, Jun 17, 2016 at 10:12 AM, Stian Soiland-Reyes <st...@apache.org>
>>> wrote:
>>>
>>>> Probably it is that the local Docker daemon has a self-signed
>>>> certificate that Java does not recognize.
>>>>
>>>>
>>>> Taverna has the credential manager - which should normally pop up and
>>>> ask if you want to accept (and store) the certificate.
>>>>
>>>> I'm not sure why that is not happening here.. perhaps because your
>>>> RESTUtil uses the Apache HTTPClient rather than the built-in URL
>>>> handling, then there is no link to the Credential Manager?
>>>>
>>>>
>>>> Could you try if it works to do a "dummy connection" first with
>>>> something like new URL("https://localhost:1239/").openConnection()
>>>> (whatever the port is) -- this should trigger the certificate
>>>> handling in Java.
>>>>
>>>>
>>>> I must admit I could not find out how this is done in the REST
>>>> activity..
>>>>
>>>> https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/RESTActivityCredentialsProvider.java
>>>>
>>>> is used for username/password (e.g. http basic auth) through Credential
>>>> Manager.
>>>>
>>>>
>>>> Does the REST activity today support https:// connections against
>>>> 'unknown' SSL certificates? (e.g. triggering the pop-ups in Workbench)
>>>>
>>>>
>>>> This file CLAIMS to do it, but doesn't seem to:
>>>>
>>>>
>>>> https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/HTTPRequestHandler.java#L116
>>>>
>>>> // Register a protocol scheme for https that uses Taverna's
>>>> // SSLSocketFactory
>>>>
>>>> but it uses the regular org.apache.http.conn.ssl.SSLSocketFactory
>>>>
>>>> .. I would have thought it should use
>>>> CredentialManager.getTavernaSSLSocketFactory() from
>>>>
>>>>
>>>> https://github.com/apache/incubator-taverna-engine/blob/master/taverna-credential-manager-impl/src/main/java/org/apache/taverna/security/credentialmanager/impl/CredentialManagerImpl.java#L2004
>>>>
>>>>
>>>> Perhaps something similar to that HTTPRequestHandler code to
>>>> initialise org.apache.http would work?
>>>>
>>>>
>>>> On 17 June 2016 at 06:32, Nadeesh Dilanga <na...@gmail.com> wrote:
>>>> > Still got stuck further on the SSL issue[1]. May be something to do
>>>> with my
>>>> > setup. So I also implemented invoking simple HTTP(without SSL) in
>>>> > additional to HTTP SSL.
>>>> >
>>>> > Meanwhile I already started getting configuration thru that spring
>>>> > inject-able framework. Will commit those changes also(without just
>>>> blocking
>>>> > on Http SSL issue) to my private repo from which I am planning to
>>>> send
>>>> > pull request.
>>>> >
>>>> > [1]:
>>>> >
>>>> > avax.net.ssl.SSLHandshakeException:
>>>> > sun.security.validator.ValidatorException: PKIX path building failed:
>>>> > sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>>> find
>>>> > valid certification path to requested target
>>>> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>>> > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1902)
>>>> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
>>>> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
>>>> > at
>>>> >
>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1338)
>>>> > at
>>>> >
>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:154)
>>>> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
>>>> > at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
>>>> > at
>>>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1032)
>>>> > at
>>>> >
>>>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1328)
>>>> > at
>>>> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
>>>> > at
>>>> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
>>>> > at
>>>> >
>>>> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
>>>> > at
>>>> >
>>>> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
>>>> > at
>>>> >
>>>> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:479)
>>>> > at
>>>> >
>>>> org.apache.http.conn.scheme.SchemeSocketFactoryAdaptor.connectSocket(SchemeSocketFactoryAdaptor.java:66)
>>>> > at
>>>> >
>>>> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
>>>> > at
>>>> >
>>>> org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
>>>> > at
>>>> >
>>>> org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
>>>> > at
>>>> >
>>>> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
>>>> > at
>>>> >
>>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
>>>> > at
>>>> >
>>>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>>>> > at
>>>> >
>>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>>>> > at
>>>> >
>>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
>>>> > at
>>>> >
>>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>>>> > at
>>>> >
>>>> org.apache.taverna.activities.docker.RESTUtil.doPost(RESTUtil.java:125)
>>>> > at
>>>> >
>>>> org.apache.taverna.activities.docker.RESTUtil.createContainer(RESTUtil.java:90)
>>>> > at
>>>> >
>>>> org.apache.taverna.activities.docker.test.TestCreateContainer.testCreateContainer(TestCreateContainer.java:40)
>>>> > at
>>>> >
>>>> org.apache.taverna.activities.docker.test.TestCreateContainer.main(TestCreateContainer.java:32)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> > at java.lang.reflect.Method.invoke(Method.java:601)
>>>> > at
>>>> com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>>>> > Caused by: sun.security.validator.ValidatorException: PKIX path
>>>> building
>>>> > failed: sun.security.provider.certpath.SunCertPathBuilderException:
>>>> unable
>>>> > to find valid certification path to requested target
>>>> > at
>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
>>>> > at
>>>> >
>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
>>>> > at sun.security.validator.Validator.validate(Validator.java:260)
>>>> > at
>>>> >
>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
>>>> > at
>>>> >
>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
>>>> > at
>>>> >
>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
>>>> > at
>>>> >
>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1320)
>>>> > ... 29 more
>>>> > Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>>> > unable to find valid certification path to requested target
>>>> > at
>>>> >
>>>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
>>>> > at
>>>> java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
>>>> > at
>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
>>>> > ... 35 more
>>>> >
>>>> >
>>>> > On Wed, Jun 15, 2016 at 6:17 PM, Nadeesh Dilanga <
>>>> nadeesh092@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> It is the container host and I am invoking the remote API. Will do
>>>> more
>>>> >> troubleshooting and get back.
>>>> >>
>>>> >> On Wed, Jun 15, 2016 at 7:05 AM, Alan Williams <
>>>> alaninmcr@googlemail.com>
>>>> >> wrote:
>>>> >>
>>>> >>> On 15-Jun-16 09:02, Nadeesh Dilanga wrote:
>>>> >>>
>>>> >>>> Hi Alan,
>>>> >>>> Thank you very much for the pointer. I went through and understood
>>>> how
>>>> >>>> works.
>>>> >>>> 1. Implement docker factory class to return an activity
>>>> configuration.
>>>> >>>> 2. Accept the configuration in the DockerActivity constructor.
>>>> >>>>
>>>> >>>> Please correct me if I missed something.
>>>> >>>>
>>>> >>>
>>>> >>> I need a picture of what is being proposed :)
>>>> >>>
>>>> >>> Also I am working on the implementation and commiting my code to my
>>>> forked
>>>> >>>> repo at [1]. I will send pull requests once following TODOs
>>>> finishes.
>>>> >>>> 1. Test HTTPS invocation(right now I am having a SSL handshake
>>>> issue and
>>>> >>>> working on it)
>>>> >>>>
>>>> >>>> I am using SSLContext.getDefault() and gives me
>>>> >>>> javax.net.ssl.SSLHandshakeException:
>>>> >>>> sun.security.validator.ValidatorException: PKIX path building
>>>> failed:
>>>> >>>> sun.security.provider.certpath.SunCertPathBuilderException: unable
>>>> to
>>>> >>>> find
>>>> >>>> valid certification path to requested target
>>>> >>>>
>>>> >>>> I have a unit test to test this. I also tried setting system
>>>> properties
>>>> >>>> javax.net.ssl.keyStore, javax.net.ssl.trustStore,
>>>> >>>> javax.net.ssl.keyStorePassword. But no luck.
>>>> >>>>
>>>> >>>
>>>> >>> What is the target of the HTTPS ? Does it have a valid certificate?
>>>> >>>
>>>> >>> 2. Enhance the activity class to read configuration injected by
>>>> Spring.
>>>> >>>>
>>>> >>>>
>>>> >>>> [1] -
>>>> >>>>
>>>> >>>>
>>>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
>>>> >>>>
>>>> >>>
>>>> >>> Great.
>>>> >>>
>>>> >>> Alan
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>
>>>>
>>>>
>>>>
>>>> --
>>>> Stian Soiland-Reyes
>>>> Apache Taverna (incubating), Apache Commons
>>>> http://orcid.org/0000-0001-9842-9718
>>>>
>>>
>>>
>>
>
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi Stain, Hi Alan,
Can you please review the code with my latest commit @
https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
where I came up with a Configuration Manager and also implemented
"Configurable".
I would like to add a unit test case for this by creating a
DockerContainerConfigurationImpl and inject a ConfigurationManager. Can you
please point me to such example. I tried, but could not find anything yet.
i.e I was not able to find a usage of "DatabaseConfigurationImpl" in
taverna-engine/taverna-osgi/taverna-common-activities repos.
Because I am planning to have a proper test case before the mid term
evaluation deadline. Also appreciate if you can provide your
feedback/expectations on this project's mid point and if it has any
required TODOs I can complete them ASAP before the mid term evaluation
deadline.
On Mon, Jun 20, 2016 at 1:47 AM, Nadeesh Dilanga <na...@gmail.com>
wrote:
> Hi all, one more thing to add.
> Create container is the most heavy API call(payload/parameter size). But
> how about things like start/stop/search/info/delete/ like methods. Should
> be think on accepting them as as activity plugin configs ?
>
>
>
>
>
>
> On Sat, Jun 18, 2016 at 3:12 AM, Nadeesh Dilanga <na...@gmail.com>
> wrote:
>
>> Hi Alan, Hi Stian,
>> Please refer my latest commit @
>>
>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker/taverna-docker-activity
>>
>> where I have implemented reading a injected configuration. Can you please
>> review this and let me know what I am missing here. But one thing I would
>> like to know is, who is responsible of creating(populating) the
>> DockerContainerConfiguration ? We have to allow user to give a docker.conf
>> and from which some one construct the DockerContainerConfiguration and
>> inject it to the activity plugin.
>>
>> Then I went through the taverna-engine repo code base looking for the
>> clue Stian gave, where I have to implement Configurable interface, and use
>> ConfigurationManager. And Configuration manager interface had
>> store/populate methods to override, but I found it bit unclear to figure
>> out how exactly I can use that to my use case/how it works/relation ship
>> between Configurable interface and ConfigurationManager. Do we have any
>> documentation on that ?
>>
>> For SSL issue, I am calling the container as
>> https://192.168.99.100:2376/containers/create where 192.168.99.100 is
>> my container host. I assume that is the target you meant ?
>>
>>
>>
>> On Fri, Jun 17, 2016 at 10:12 AM, Stian Soiland-Reyes <st...@apache.org>
>> wrote:
>>
>>> Probably it is that the local Docker daemon has a self-signed
>>> certificate that Java does not recognize.
>>>
>>>
>>> Taverna has the credential manager - which should normally pop up and
>>> ask if you want to accept (and store) the certificate.
>>>
>>> I'm not sure why that is not happening here.. perhaps because your
>>> RESTUtil uses the Apache HTTPClient rather than the built-in URL
>>> handling, then there is no link to the Credential Manager?
>>>
>>>
>>> Could you try if it works to do a "dummy connection" first with
>>> something like new URL("https://localhost:1239/").openConnection()
>>> (whatever the port is) -- this should trigger the certificate
>>> handling in Java.
>>>
>>>
>>> I must admit I could not find out how this is done in the REST activity..
>>>
>>> https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/RESTActivityCredentialsProvider.java
>>>
>>> is used for username/password (e.g. http basic auth) through Credential
>>> Manager.
>>>
>>>
>>> Does the REST activity today support https:// connections against
>>> 'unknown' SSL certificates? (e.g. triggering the pop-ups in Workbench)
>>>
>>>
>>> This file CLAIMS to do it, but doesn't seem to:
>>>
>>>
>>> https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/HTTPRequestHandler.java#L116
>>>
>>> // Register a protocol scheme for https that uses Taverna's
>>> // SSLSocketFactory
>>>
>>> but it uses the regular org.apache.http.conn.ssl.SSLSocketFactory
>>>
>>> .. I would have thought it should use
>>> CredentialManager.getTavernaSSLSocketFactory() from
>>>
>>>
>>> https://github.com/apache/incubator-taverna-engine/blob/master/taverna-credential-manager-impl/src/main/java/org/apache/taverna/security/credentialmanager/impl/CredentialManagerImpl.java#L2004
>>>
>>>
>>> Perhaps something similar to that HTTPRequestHandler code to
>>> initialise org.apache.http would work?
>>>
>>>
>>> On 17 June 2016 at 06:32, Nadeesh Dilanga <na...@gmail.com> wrote:
>>> > Still got stuck further on the SSL issue[1]. May be something to do
>>> with my
>>> > setup. So I also implemented invoking simple HTTP(without SSL) in
>>> > additional to HTTP SSL.
>>> >
>>> > Meanwhile I already started getting configuration thru that spring
>>> > inject-able framework. Will commit those changes also(without just
>>> blocking
>>> > on Http SSL issue) to my private repo from which I am planning to send
>>> > pull request.
>>> >
>>> > [1]:
>>> >
>>> > avax.net.ssl.SSLHandshakeException:
>>> > sun.security.validator.ValidatorException: PKIX path building failed:
>>> > sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>> find
>>> > valid certification path to requested target
>>> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>> > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1902)
>>> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
>>> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
>>> > at
>>> >
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1338)
>>> > at
>>> >
>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:154)
>>> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
>>> > at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
>>> > at
>>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1032)
>>> > at
>>> >
>>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1328)
>>> > at
>>> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
>>> > at
>>> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
>>> > at
>>> >
>>> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
>>> > at
>>> >
>>> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
>>> > at
>>> >
>>> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:479)
>>> > at
>>> >
>>> org.apache.http.conn.scheme.SchemeSocketFactoryAdaptor.connectSocket(SchemeSocketFactoryAdaptor.java:66)
>>> > at
>>> >
>>> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
>>> > at
>>> >
>>> org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
>>> > at
>>> >
>>> org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
>>> > at
>>> >
>>> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
>>> > at
>>> >
>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
>>> > at
>>> >
>>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>>> > at
>>> >
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>>> > at
>>> >
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
>>> > at
>>> >
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>>> > at
>>> > org.apache.taverna.activities.docker.RESTUtil.doPost(RESTUtil.java:125)
>>> > at
>>> >
>>> org.apache.taverna.activities.docker.RESTUtil.createContainer(RESTUtil.java:90)
>>> > at
>>> >
>>> org.apache.taverna.activities.docker.test.TestCreateContainer.testCreateContainer(TestCreateContainer.java:40)
>>> > at
>>> >
>>> org.apache.taverna.activities.docker.test.TestCreateContainer.main(TestCreateContainer.java:32)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > at java.lang.reflect.Method.invoke(Method.java:601)
>>> > at
>>> com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>>> > Caused by: sun.security.validator.ValidatorException: PKIX path
>>> building
>>> > failed: sun.security.provider.certpath.SunCertPathBuilderException:
>>> unable
>>> > to find valid certification path to requested target
>>> > at
>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
>>> > at
>>> >
>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
>>> > at sun.security.validator.Validator.validate(Validator.java:260)
>>> > at
>>> >
>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
>>> > at
>>> >
>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
>>> > at
>>> >
>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
>>> > at
>>> >
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1320)
>>> > ... 29 more
>>> > Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>> > unable to find valid certification path to requested target
>>> > at
>>> >
>>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
>>> > at
>>> java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
>>> > at
>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
>>> > ... 35 more
>>> >
>>> >
>>> > On Wed, Jun 15, 2016 at 6:17 PM, Nadeesh Dilanga <nadeesh092@gmail.com
>>> >
>>> > wrote:
>>> >
>>> >> It is the container host and I am invoking the remote API. Will do
>>> more
>>> >> troubleshooting and get back.
>>> >>
>>> >> On Wed, Jun 15, 2016 at 7:05 AM, Alan Williams <
>>> alaninmcr@googlemail.com>
>>> >> wrote:
>>> >>
>>> >>> On 15-Jun-16 09:02, Nadeesh Dilanga wrote:
>>> >>>
>>> >>>> Hi Alan,
>>> >>>> Thank you very much for the pointer. I went through and understood
>>> how
>>> >>>> works.
>>> >>>> 1. Implement docker factory class to return an activity
>>> configuration.
>>> >>>> 2. Accept the configuration in the DockerActivity constructor.
>>> >>>>
>>> >>>> Please correct me if I missed something.
>>> >>>>
>>> >>>
>>> >>> I need a picture of what is being proposed :)
>>> >>>
>>> >>> Also I am working on the implementation and commiting my code to my
>>> forked
>>> >>>> repo at [1]. I will send pull requests once following TODOs
>>> finishes.
>>> >>>> 1. Test HTTPS invocation(right now I am having a SSL handshake
>>> issue and
>>> >>>> working on it)
>>> >>>>
>>> >>>> I am using SSLContext.getDefault() and gives me
>>> >>>> javax.net.ssl.SSLHandshakeException:
>>> >>>> sun.security.validator.ValidatorException: PKIX path building
>>> failed:
>>> >>>> sun.security.provider.certpath.SunCertPathBuilderException: unable
>>> to
>>> >>>> find
>>> >>>> valid certification path to requested target
>>> >>>>
>>> >>>> I have a unit test to test this. I also tried setting system
>>> properties
>>> >>>> javax.net.ssl.keyStore, javax.net.ssl.trustStore,
>>> >>>> javax.net.ssl.keyStorePassword. But no luck.
>>> >>>>
>>> >>>
>>> >>> What is the target of the HTTPS ? Does it have a valid certificate?
>>> >>>
>>> >>> 2. Enhance the activity class to read configuration injected by
>>> Spring.
>>> >>>>
>>> >>>>
>>> >>>> [1] -
>>> >>>>
>>> >>>>
>>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
>>> >>>>
>>> >>>
>>> >>> Great.
>>> >>>
>>> >>> Alan
>>> >>>
>>> >>>
>>> >>>
>>> >>
>>>
>>>
>>>
>>> --
>>> Stian Soiland-Reyes
>>> Apache Taverna (incubating), Apache Commons
>>> http://orcid.org/0000-0001-9842-9718
>>>
>>
>>
>
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi all, one more thing to add.
Create container is the most heavy API call(payload/parameter size). But
how about things like start/stop/search/info/delete/ like methods. Should
be think on accepting them as as activity plugin configs ?
On Sat, Jun 18, 2016 at 3:12 AM, Nadeesh Dilanga <na...@gmail.com>
wrote:
> Hi Alan, Hi Stian,
> Please refer my latest commit @
>
> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker/taverna-docker-activity
>
> where I have implemented reading a injected configuration. Can you please
> review this and let me know what I am missing here. But one thing I would
> like to know is, who is responsible of creating(populating) the
> DockerContainerConfiguration ? We have to allow user to give a docker.conf
> and from which some one construct the DockerContainerConfiguration and
> inject it to the activity plugin.
>
> Then I went through the taverna-engine repo code base looking for the clue
> Stian gave, where I have to implement Configurable interface, and use
> ConfigurationManager. And Configuration manager interface had
> store/populate methods to override, but I found it bit unclear to figure
> out how exactly I can use that to my use case/how it works/relation ship
> between Configurable interface and ConfigurationManager. Do we have any
> documentation on that ?
>
> For SSL issue, I am calling the container as
> https://192.168.99.100:2376/containers/create where 192.168.99.100 is my
> container host. I assume that is the target you meant ?
>
>
>
> On Fri, Jun 17, 2016 at 10:12 AM, Stian Soiland-Reyes <st...@apache.org>
> wrote:
>
>> Probably it is that the local Docker daemon has a self-signed
>> certificate that Java does not recognize.
>>
>>
>> Taverna has the credential manager - which should normally pop up and
>> ask if you want to accept (and store) the certificate.
>>
>> I'm not sure why that is not happening here.. perhaps because your
>> RESTUtil uses the Apache HTTPClient rather than the built-in URL
>> handling, then there is no link to the Credential Manager?
>>
>>
>> Could you try if it works to do a "dummy connection" first with
>> something like new URL("https://localhost:1239/").openConnection()
>> (whatever the port is) -- this should trigger the certificate
>> handling in Java.
>>
>>
>> I must admit I could not find out how this is done in the REST activity..
>>
>> https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/RESTActivityCredentialsProvider.java
>>
>> is used for username/password (e.g. http basic auth) through Credential
>> Manager.
>>
>>
>> Does the REST activity today support https:// connections against
>> 'unknown' SSL certificates? (e.g. triggering the pop-ups in Workbench)
>>
>>
>> This file CLAIMS to do it, but doesn't seem to:
>>
>>
>> https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/HTTPRequestHandler.java#L116
>>
>> // Register a protocol scheme for https that uses Taverna's
>> // SSLSocketFactory
>>
>> but it uses the regular org.apache.http.conn.ssl.SSLSocketFactory
>>
>> .. I would have thought it should use
>> CredentialManager.getTavernaSSLSocketFactory() from
>>
>>
>> https://github.com/apache/incubator-taverna-engine/blob/master/taverna-credential-manager-impl/src/main/java/org/apache/taverna/security/credentialmanager/impl/CredentialManagerImpl.java#L2004
>>
>>
>> Perhaps something similar to that HTTPRequestHandler code to
>> initialise org.apache.http would work?
>>
>>
>> On 17 June 2016 at 06:32, Nadeesh Dilanga <na...@gmail.com> wrote:
>> > Still got stuck further on the SSL issue[1]. May be something to do
>> with my
>> > setup. So I also implemented invoking simple HTTP(without SSL) in
>> > additional to HTTP SSL.
>> >
>> > Meanwhile I already started getting configuration thru that spring
>> > inject-able framework. Will commit those changes also(without just
>> blocking
>> > on Http SSL issue) to my private repo from which I am planning to send
>> > pull request.
>> >
>> > [1]:
>> >
>> > avax.net.ssl.SSLHandshakeException:
>> > sun.security.validator.ValidatorException: PKIX path building failed:
>> > sun.security.provider.certpath.SunCertPathBuilderException: unable to
>> find
>> > valid certification path to requested target
>> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>> > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1902)
>> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
>> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
>> > at
>> >
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1338)
>> > at
>> >
>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:154)
>> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
>> > at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
>> > at
>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1032)
>> > at
>> >
>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1328)
>> > at
>> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
>> > at
>> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
>> > at
>> >
>> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
>> > at
>> >
>> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
>> > at
>> >
>> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:479)
>> > at
>> >
>> org.apache.http.conn.scheme.SchemeSocketFactoryAdaptor.connectSocket(SchemeSocketFactoryAdaptor.java:66)
>> > at
>> >
>> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
>> > at
>> >
>> org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
>> > at
>> >
>> org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
>> > at
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
>> > at
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
>> > at
>> >
>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>> > at
>> >
>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>> > at
>> >
>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
>> > at
>> >
>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>> > at
>> > org.apache.taverna.activities.docker.RESTUtil.doPost(RESTUtil.java:125)
>> > at
>> >
>> org.apache.taverna.activities.docker.RESTUtil.createContainer(RESTUtil.java:90)
>> > at
>> >
>> org.apache.taverna.activities.docker.test.TestCreateContainer.testCreateContainer(TestCreateContainer.java:40)
>> > at
>> >
>> org.apache.taverna.activities.docker.test.TestCreateContainer.main(TestCreateContainer.java:32)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:601)
>> > at
>> com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>> > Caused by: sun.security.validator.ValidatorException: PKIX path building
>> > failed: sun.security.provider.certpath.SunCertPathBuilderException:
>> unable
>> > to find valid certification path to requested target
>> > at
>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
>> > at
>> >
>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
>> > at sun.security.validator.Validator.validate(Validator.java:260)
>> > at
>> >
>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
>> > at
>> >
>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
>> > at
>> >
>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
>> > at
>> >
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1320)
>> > ... 29 more
>> > Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>> > unable to find valid certification path to requested target
>> > at
>> >
>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
>> > at
>> java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
>> > at
>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
>> > ... 35 more
>> >
>> >
>> > On Wed, Jun 15, 2016 at 6:17 PM, Nadeesh Dilanga <na...@gmail.com>
>> > wrote:
>> >
>> >> It is the container host and I am invoking the remote API. Will do more
>> >> troubleshooting and get back.
>> >>
>> >> On Wed, Jun 15, 2016 at 7:05 AM, Alan Williams <
>> alaninmcr@googlemail.com>
>> >> wrote:
>> >>
>> >>> On 15-Jun-16 09:02, Nadeesh Dilanga wrote:
>> >>>
>> >>>> Hi Alan,
>> >>>> Thank you very much for the pointer. I went through and understood
>> how
>> >>>> works.
>> >>>> 1. Implement docker factory class to return an activity
>> configuration.
>> >>>> 2. Accept the configuration in the DockerActivity constructor.
>> >>>>
>> >>>> Please correct me if I missed something.
>> >>>>
>> >>>
>> >>> I need a picture of what is being proposed :)
>> >>>
>> >>> Also I am working on the implementation and commiting my code to my
>> forked
>> >>>> repo at [1]. I will send pull requests once following TODOs finishes.
>> >>>> 1. Test HTTPS invocation(right now I am having a SSL handshake
>> issue and
>> >>>> working on it)
>> >>>>
>> >>>> I am using SSLContext.getDefault() and gives me
>> >>>> javax.net.ssl.SSLHandshakeException:
>> >>>> sun.security.validator.ValidatorException: PKIX path building failed:
>> >>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>> >>>> find
>> >>>> valid certification path to requested target
>> >>>>
>> >>>> I have a unit test to test this. I also tried setting system
>> properties
>> >>>> javax.net.ssl.keyStore, javax.net.ssl.trustStore,
>> >>>> javax.net.ssl.keyStorePassword. But no luck.
>> >>>>
>> >>>
>> >>> What is the target of the HTTPS ? Does it have a valid certificate?
>> >>>
>> >>> 2. Enhance the activity class to read configuration injected by
>> Spring.
>> >>>>
>> >>>>
>> >>>> [1] -
>> >>>>
>> >>>>
>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
>> >>>>
>> >>>
>> >>> Great.
>> >>>
>> >>> Alan
>> >>>
>> >>>
>> >>>
>> >>
>>
>>
>>
>> --
>> Stian Soiland-Reyes
>> Apache Taverna (incubating), Apache Commons
>> http://orcid.org/0000-0001-9842-9718
>>
>
>
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi Alan, Hi Stian,
Please refer my latest commit @
https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker/taverna-docker-activity
where I have implemented reading a injected configuration. Can you please
review this and let me know what I am missing here. But one thing I would
like to know is, who is responsible of creating(populating) the
DockerContainerConfiguration ? We have to allow user to give a docker.conf
and from which some one construct the DockerContainerConfiguration and
inject it to the activity plugin.
Then I went through the taverna-engine repo code base looking for the clue
Stian gave, where I have to implement Configurable interface, and use
ConfigurationManager. And Configuration manager interface had
store/populate methods to override, but I found it bit unclear to figure
out how exactly I can use that to my use case/how it works/relation ship
between Configurable interface and ConfigurationManager. Do we have any
documentation on that ?
For SSL issue, I am calling the container as
https://192.168.99.100:2376/containers/create where 192.168.99.100 is my
container host. I assume that is the target you meant ?
On Fri, Jun 17, 2016 at 10:12 AM, Stian Soiland-Reyes <st...@apache.org>
wrote:
> Probably it is that the local Docker daemon has a self-signed
> certificate that Java does not recognize.
>
>
> Taverna has the credential manager - which should normally pop up and
> ask if you want to accept (and store) the certificate.
>
> I'm not sure why that is not happening here.. perhaps because your
> RESTUtil uses the Apache HTTPClient rather than the built-in URL
> handling, then there is no link to the Credential Manager?
>
>
> Could you try if it works to do a "dummy connection" first with
> something like new URL("https://localhost:1239/").openConnection()
> (whatever the port is) -- this should trigger the certificate
> handling in Java.
>
>
> I must admit I could not find out how this is done in the REST activity..
>
> https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/RESTActivityCredentialsProvider.java
>
> is used for username/password (e.g. http basic auth) through Credential
> Manager.
>
>
> Does the REST activity today support https:// connections against
> 'unknown' SSL certificates? (e.g. triggering the pop-ups in Workbench)
>
>
> This file CLAIMS to do it, but doesn't seem to:
>
>
> https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/HTTPRequestHandler.java#L116
>
> // Register a protocol scheme for https that uses Taverna's
> // SSLSocketFactory
>
> but it uses the regular org.apache.http.conn.ssl.SSLSocketFactory
>
> .. I would have thought it should use
> CredentialManager.getTavernaSSLSocketFactory() from
>
>
> https://github.com/apache/incubator-taverna-engine/blob/master/taverna-credential-manager-impl/src/main/java/org/apache/taverna/security/credentialmanager/impl/CredentialManagerImpl.java#L2004
>
>
> Perhaps something similar to that HTTPRequestHandler code to
> initialise org.apache.http would work?
>
>
> On 17 June 2016 at 06:32, Nadeesh Dilanga <na...@gmail.com> wrote:
> > Still got stuck further on the SSL issue[1]. May be something to do with
> my
> > setup. So I also implemented invoking simple HTTP(without SSL) in
> > additional to HTTP SSL.
> >
> > Meanwhile I already started getting configuration thru that spring
> > inject-able framework. Will commit those changes also(without just
> blocking
> > on Http SSL issue) to my private repo from which I am planning to send
> > pull request.
> >
> > [1]:
> >
> > avax.net.ssl.SSLHandshakeException:
> > sun.security.validator.ValidatorException: PKIX path building failed:
> > sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find
> > valid certification path to requested target
> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1902)
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
> > at
> >
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1338)
> > at
> >
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:154)
> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
> > at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
> > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1032)
> > at
> >
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1328)
> > at
> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
> > at
> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
> > at
> >
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> > at
> >
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> > at
> >
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:479)
> > at
> >
> org.apache.http.conn.scheme.SchemeSocketFactoryAdaptor.connectSocket(SchemeSocketFactoryAdaptor.java:66)
> > at
> >
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> > at
> >
> org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
> > at
> >
> org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
> > at
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> > at
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> > at
> >
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> > at
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> > at
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> > at
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> > at
> > org.apache.taverna.activities.docker.RESTUtil.doPost(RESTUtil.java:125)
> > at
> >
> org.apache.taverna.activities.docker.RESTUtil.createContainer(RESTUtil.java:90)
> > at
> >
> org.apache.taverna.activities.docker.test.TestCreateContainer.testCreateContainer(TestCreateContainer.java:40)
> > at
> >
> org.apache.taverna.activities.docker.test.TestCreateContainer.main(TestCreateContainer.java:32)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:601)
> > at
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> > Caused by: sun.security.validator.ValidatorException: PKIX path building
> > failed: sun.security.provider.certpath.SunCertPathBuilderException:
> unable
> > to find valid certification path to requested target
> > at
> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
> > at
> >
> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
> > at sun.security.validator.Validator.validate(Validator.java:260)
> > at
> >
> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
> > at
> >
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
> > at
> >
> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
> > at
> >
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1320)
> > ... 29 more
> > Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
> > unable to find valid certification path to requested target
> > at
> >
> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
> > at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
> > at
> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
> > ... 35 more
> >
> >
> > On Wed, Jun 15, 2016 at 6:17 PM, Nadeesh Dilanga <na...@gmail.com>
> > wrote:
> >
> >> It is the container host and I am invoking the remote API. Will do more
> >> troubleshooting and get back.
> >>
> >> On Wed, Jun 15, 2016 at 7:05 AM, Alan Williams <
> alaninmcr@googlemail.com>
> >> wrote:
> >>
> >>> On 15-Jun-16 09:02, Nadeesh Dilanga wrote:
> >>>
> >>>> Hi Alan,
> >>>> Thank you very much for the pointer. I went through and understood how
> >>>> works.
> >>>> 1. Implement docker factory class to return an activity configuration.
> >>>> 2. Accept the configuration in the DockerActivity constructor.
> >>>>
> >>>> Please correct me if I missed something.
> >>>>
> >>>
> >>> I need a picture of what is being proposed :)
> >>>
> >>> Also I am working on the implementation and commiting my code to my
> forked
> >>>> repo at [1]. I will send pull requests once following TODOs finishes.
> >>>> 1. Test HTTPS invocation(right now I am having a SSL handshake issue
> and
> >>>> working on it)
> >>>>
> >>>> I am using SSLContext.getDefault() and gives me
> >>>> javax.net.ssl.SSLHandshakeException:
> >>>> sun.security.validator.ValidatorException: PKIX path building failed:
> >>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
> >>>> find
> >>>> valid certification path to requested target
> >>>>
> >>>> I have a unit test to test this. I also tried setting system
> properties
> >>>> javax.net.ssl.keyStore, javax.net.ssl.trustStore,
> >>>> javax.net.ssl.keyStorePassword. But no luck.
> >>>>
> >>>
> >>> What is the target of the HTTPS ? Does it have a valid certificate?
> >>>
> >>> 2. Enhance the activity class to read configuration injected by
> Spring.
> >>>>
> >>>>
> >>>> [1] -
> >>>>
> >>>>
> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
> >>>>
> >>>
> >>> Great.
> >>>
> >>> Alan
> >>>
> >>>
> >>>
> >>
>
>
>
> --
> Stian Soiland-Reyes
> Apache Taverna (incubating), Apache Commons
> http://orcid.org/0000-0001-9842-9718
>
Re: Finalize Docker Invoke JSON format
Posted by Stian Soiland-Reyes <st...@apache.org>.
Probably it is that the local Docker daemon has a self-signed
certificate that Java does not recognize.
Taverna has the credential manager - which should normally pop up and
ask if you want to accept (and store) the certificate.
I'm not sure why that is not happening here.. perhaps because your
RESTUtil uses the Apache HTTPClient rather than the built-in URL
handling, then there is no link to the Credential Manager?
Could you try if it works to do a "dummy connection" first with
something like new URL("https://localhost:1239/").openConnection()
(whatever the port is) -- this should trigger the certificate
handling in Java.
I must admit I could not find out how this is done in the REST activity..
https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/RESTActivityCredentialsProvider.java
is used for username/password (e.g. http basic auth) through Credential Manager.
Does the REST activity today support https:// connections against
'unknown' SSL certificates? (e.g. triggering the pop-ups in Workbench)
This file CLAIMS to do it, but doesn't seem to:
https://github.com/apache/incubator-taverna-common-activities/blob/master/taverna-rest-activity/src/main/java/org/apache/taverna/activities/rest/HTTPRequestHandler.java#L116
// Register a protocol scheme for https that uses Taverna's
// SSLSocketFactory
but it uses the regular org.apache.http.conn.ssl.SSLSocketFactory
.. I would have thought it should use
CredentialManager.getTavernaSSLSocketFactory() from
https://github.com/apache/incubator-taverna-engine/blob/master/taverna-credential-manager-impl/src/main/java/org/apache/taverna/security/credentialmanager/impl/CredentialManagerImpl.java#L2004
Perhaps something similar to that HTTPRequestHandler code to
initialise org.apache.http would work?
On 17 June 2016 at 06:32, Nadeesh Dilanga <na...@gmail.com> wrote:
> Still got stuck further on the SSL issue[1]. May be something to do with my
> setup. So I also implemented invoking simple HTTP(without SSL) in
> additional to HTTP SSL.
>
> Meanwhile I already started getting configuration thru that spring
> inject-able framework. Will commit those changes also(without just blocking
> on Http SSL issue) to my private repo from which I am planning to send
> pull request.
>
> [1]:
>
> avax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1902)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1338)
> at
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:154)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
> at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1032)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1328)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
> at
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> at
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> at
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:479)
> at
> org.apache.http.conn.scheme.SchemeSocketFactoryAdaptor.connectSocket(SchemeSocketFactoryAdaptor.java:66)
> at
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> at
> org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
> at
> org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
> at
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
> org.apache.taverna.activities.docker.RESTUtil.doPost(RESTUtil.java:125)
> at
> org.apache.taverna.activities.docker.RESTUtil.createContainer(RESTUtil.java:90)
> at
> org.apache.taverna.activities.docker.test.TestCreateContainer.testCreateContainer(TestCreateContainer.java:40)
> at
> org.apache.taverna.activities.docker.test.TestCreateContainer.main(TestCreateContainer.java:32)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> Caused by: sun.security.validator.ValidatorException: PKIX path building
> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
> to find valid certification path to requested target
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
> at
> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
> at sun.security.validator.Validator.validate(Validator.java:260)
> at
> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
> at
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
> at
> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1320)
> ... 29 more
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
> unable to find valid certification path to requested target
> at
> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
> ... 35 more
>
>
> On Wed, Jun 15, 2016 at 6:17 PM, Nadeesh Dilanga <na...@gmail.com>
> wrote:
>
>> It is the container host and I am invoking the remote API. Will do more
>> troubleshooting and get back.
>>
>> On Wed, Jun 15, 2016 at 7:05 AM, Alan Williams <al...@googlemail.com>
>> wrote:
>>
>>> On 15-Jun-16 09:02, Nadeesh Dilanga wrote:
>>>
>>>> Hi Alan,
>>>> Thank you very much for the pointer. I went through and understood how
>>>> works.
>>>> 1. Implement docker factory class to return an activity configuration.
>>>> 2. Accept the configuration in the DockerActivity constructor.
>>>>
>>>> Please correct me if I missed something.
>>>>
>>>
>>> I need a picture of what is being proposed :)
>>>
>>> Also I am working on the implementation and commiting my code to my forked
>>>> repo at [1]. I will send pull requests once following TODOs finishes.
>>>> 1. Test HTTPS invocation(right now I am having a SSL handshake issue and
>>>> working on it)
>>>>
>>>> I am using SSLContext.getDefault() and gives me
>>>> javax.net.ssl.SSLHandshakeException:
>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>>> find
>>>> valid certification path to requested target
>>>>
>>>> I have a unit test to test this. I also tried setting system properties
>>>> javax.net.ssl.keyStore, javax.net.ssl.trustStore,
>>>> javax.net.ssl.keyStorePassword. But no luck.
>>>>
>>>
>>> What is the target of the HTTPS ? Does it have a valid certificate?
>>>
>>> 2. Enhance the activity class to read configuration injected by Spring.
>>>>
>>>>
>>>> [1] -
>>>>
>>>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
>>>>
>>>
>>> Great.
>>>
>>> Alan
>>>
>>>
>>>
>>
--
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons
http://orcid.org/0000-0001-9842-9718
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 17-Jun-16 05:32, Nadeesh Dilanga wrote:
> Still got stuck further on the SSL issue[1]. May be something to do with my
> setup. So I also implemented invoking simple HTTP(without SSL) in
> additional to HTTP SSL.
> Meanwhile I already started getting configuration thru that spring
> inject-able framework. Will commit those changes also(without just blocking
> on Http SSL issue) to my private repo from which I am planning to send
> pull request.
Great. Can you please do that as soon as possible?
> [1]:
>
> avax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
What is the target? Are you sure there is actually a certification path
to the target?
Alan
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Still got stuck further on the SSL issue[1]. May be something to do with my
setup. So I also implemented invoking simple HTTP(without SSL) in
additional to HTTP SSL.
Meanwhile I already started getting configuration thru that spring
inject-able framework. Will commit those changes also(without just blocking
on Http SSL issue) to my private repo from which I am planning to send
pull request.
[1]:
avax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1902)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1338)
at
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:154)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1032)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1328)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at
org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
at
org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
at
org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:479)
at
org.apache.http.conn.scheme.SchemeSocketFactoryAdaptor.connectSocket(SchemeSocketFactoryAdaptor.java:66)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
at
org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at
org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.taverna.activities.docker.RESTUtil.doPost(RESTUtil.java:125)
at
org.apache.taverna.activities.docker.RESTUtil.createContainer(RESTUtil.java:90)
at
org.apache.taverna.activities.docker.test.TestCreateContainer.testCreateContainer(TestCreateContainer.java:40)
at
org.apache.taverna.activities.docker.test.TestCreateContainer.main(TestCreateContainer.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: sun.security.validator.ValidatorException: PKIX path building
failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at
sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at
sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
at
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
at
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1320)
... 29 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target
at
sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
... 35 more
On Wed, Jun 15, 2016 at 6:17 PM, Nadeesh Dilanga <na...@gmail.com>
wrote:
> It is the container host and I am invoking the remote API. Will do more
> troubleshooting and get back.
>
> On Wed, Jun 15, 2016 at 7:05 AM, Alan Williams <al...@googlemail.com>
> wrote:
>
>> On 15-Jun-16 09:02, Nadeesh Dilanga wrote:
>>
>>> Hi Alan,
>>> Thank you very much for the pointer. I went through and understood how
>>> works.
>>> 1. Implement docker factory class to return an activity configuration.
>>> 2. Accept the configuration in the DockerActivity constructor.
>>>
>>> Please correct me if I missed something.
>>>
>>
>> I need a picture of what is being proposed :)
>>
>> Also I am working on the implementation and commiting my code to my forked
>>> repo at [1]. I will send pull requests once following TODOs finishes.
>>> 1. Test HTTPS invocation(right now I am having a SSL handshake issue and
>>> working on it)
>>>
>>> I am using SSLContext.getDefault() and gives me
>>> javax.net.ssl.SSLHandshakeException:
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>> find
>>> valid certification path to requested target
>>>
>>> I have a unit test to test this. I also tried setting system properties
>>> javax.net.ssl.keyStore, javax.net.ssl.trustStore,
>>> javax.net.ssl.keyStorePassword. But no luck.
>>>
>>
>> What is the target of the HTTPS ? Does it have a valid certificate?
>>
>> 2. Enhance the activity class to read configuration injected by Spring.
>>>
>>>
>>> [1] -
>>>
>>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
>>>
>>
>> Great.
>>
>> Alan
>>
>>
>>
>
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
It is the container host and I am invoking the remote API. Will do more
troubleshooting and get back.
On Wed, Jun 15, 2016 at 7:05 AM, Alan Williams <al...@googlemail.com>
wrote:
> On 15-Jun-16 09:02, Nadeesh Dilanga wrote:
>
>> Hi Alan,
>> Thank you very much for the pointer. I went through and understood how
>> works.
>> 1. Implement docker factory class to return an activity configuration.
>> 2. Accept the configuration in the DockerActivity constructor.
>>
>> Please correct me if I missed something.
>>
>
> I need a picture of what is being proposed :)
>
> Also I am working on the implementation and commiting my code to my forked
>> repo at [1]. I will send pull requests once following TODOs finishes.
>> 1. Test HTTPS invocation(right now I am having a SSL handshake issue and
>> working on it)
>>
>> I am using SSLContext.getDefault() and gives me
>> javax.net.ssl.SSLHandshakeException:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target
>>
>> I have a unit test to test this. I also tried setting system properties
>> javax.net.ssl.keyStore, javax.net.ssl.trustStore,
>> javax.net.ssl.keyStorePassword. But no luck.
>>
>
> What is the target of the HTTPS ? Does it have a valid certificate?
>
> 2. Enhance the activity class to read configuration injected by Spring.
>>
>>
>> [1] -
>>
>> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
>>
>
> Great.
>
> Alan
>
>
>
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 15-Jun-16 09:02, Nadeesh Dilanga wrote:
> Hi Alan,
> Thank you very much for the pointer. I went through and understood how
> works.
> 1. Implement docker factory class to return an activity configuration.
> 2. Accept the configuration in the DockerActivity constructor.
>
> Please correct me if I missed something.
I need a picture of what is being proposed :)
> Also I am working on the implementation and commiting my code to my forked
> repo at [1]. I will send pull requests once following TODOs finishes.
> 1. Test HTTPS invocation(right now I am having a SSL handshake issue and
> working on it)
>
> I am using SSLContext.getDefault() and gives me
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
>
> I have a unit test to test this. I also tried setting system properties
> javax.net.ssl.keyStore, javax.net.ssl.trustStore,
> javax.net.ssl.keyStorePassword. But no luck.
What is the target of the HTTPS ? Does it have a valid certificate?
> 2. Enhance the activity class to read configuration injected by Spring.
>
>
> [1] -
> https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
Great.
Alan
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi Alan,
Thank you very much for the pointer. I went through and understood how
works.
1. Implement docker factory class to return an activity configuration.
2. Accept the configuration in the DockerActivity constructor.
Please correct me if I missed something.
Also I am working on the implementation and commiting my code to my forked
repo at [1]. I will send pull requests once following TODOs finishes.
1. Test HTTPS invocation(right now I am having a SSL handshake issue and
working on it)
I am using SSLContext.getDefault() and gives me
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to requested target
I have a unit test to test this. I also tried setting system properties
javax.net.ssl.keyStore, javax.net.ssl.trustStore,
javax.net.ssl.keyStorePassword. But no luck.
2. Enhance the activity class to read configuration injected by Spring.
[1] -
https://github.com/NadeeshDilanga/incubator-taverna-common-activities/commits/docker
On Tue, Jun 14, 2016 at 5:06 AM, Alan Williams <al...@googlemail.com>
wrote:
> On 14-Jun-16 05:04, Nadeesh Dilanga wrote:
>
>> Hi Alan,
>> Do you have an example link where we have used such Spring Configuration ?
>>
>
> Firstly, just to check are you targeting Taverna 2 or Taverna 3?
>
> The Spring Configuration is for Taverna 3. You can see an example in
> /taverna-rest-activity/src/main/resources/META-INF/spring/rest-activity-context-osgi.xml
> where the credentialManager is "pulled in" by
>
> <reference id="credentialManager"
> interface="org.apache.taverna.security.credentialmanager.CredentialManager"
> />
>
> The credentialManager was "pushed" by the Taverna engine in
>
>
> taverna-credential-manager-impl/src/main/resources/META-INF/spring/credential-manager-impl-context.xml
>
> Alan
>
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 14-Jun-16 05:04, Nadeesh Dilanga wrote:
> Hi Alan,
> Do you have an example link where we have used such Spring Configuration ?
Firstly, just to check are you targeting Taverna 2 or Taverna 3?
The Spring Configuration is for Taverna 3. You can see an example in
/taverna-rest-activity/src/main/resources/META-INF/spring/rest-activity-context-osgi.xml
where the credentialManager is "pulled in" by
<reference id="credentialManager"
interface="org.apache.taverna.security.credentialmanager.CredentialManager"
/>
The credentialManager was "pushed" by the Taverna engine in
taverna-credential-manager-impl/src/main/resources/META-INF/spring/credential-manager-impl-context.xml
Alan
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi Alan,
Do you have an example link where we have used such Spring Configuration ?
On Mon, Jun 13, 2016 at 9:28 AM, Alan Williams <al...@googlemail.com>
wrote:
> On 10-Jun-16 02:49, Nadeesh Dilanga wrote:
>
>> Hi Stian,
>>
>> Following API can be used to attach to a remote Docker. Refer "Attach to
>> a
>> container" for more details.
>>
>> POST /containers/(id or name)/attach
>>
>> Ok, it make sense to read it from some sort of a configuration instead
>> from
>> the input JSON for the activity plugin. Inside an activity plugin, how can
>> I access this configuration ?
>>
>
> The activity plugin has a DockerContainerConfiguration injected by the
> Spring configuration. You would need a setDockerContainerConfigfuration
> method. For simplicity, you can have the dockerContainerConfiguration field
> as static.
>
> @Stian - does that sound reasonable ?
>
> Alan
>
>
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 10-Jun-16 02:49, Nadeesh Dilanga wrote:
> Hi Stian,
>
> Following API can be used to attach to a remote Docker. Refer "Attach to a
> container" for more details.
>
> POST /containers/(id or name)/attach
>
> Ok, it make sense to read it from some sort of a configuration instead from
> the input JSON for the activity plugin. Inside an activity plugin, how can
> I access this configuration ?
The activity plugin has a DockerContainerConfiguration injected by the
Spring configuration. You would need a setDockerContainerConfigfuration
method. For simplicity, you can have the dockerContainerConfiguration
field as static.
@Stian - does that sound reasonable ?
Alan
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi Stian,
Following API can be used to attach to a remote Docker. Refer "Attach to a
container" for more details.
POST /containers/(id or name)/attach
Ok, it make sense to read it from some sort of a configuration instead from
the input JSON for the activity plugin. Inside an activity plugin, how can
I access this configuration ?
On Wed, Jun 8, 2016 at 12:41 PM, Stian Soiland-Reyes <st...@apache.org>
wrote:
> Perhaps we need a configuration/preferences per Workbench/workflow
> about where to run the docker images. It's going to be different per
> user.. e.g. if you use Docker on a Mac then it will be some local IP
> address to the virtual machine - but then it's probably going to be
> the same for each workflow, or for each step in the same workflow.
>
> So do you think we need it to be part of the Activity configuration?
> Perhaps unnecessary flexibility..
>
>
>
> I don't think we need to aim for a Docker Machine-type or Docker
> Swarm-type scenario, although that would be cool to keep in mind to
> enable cloud-based workflows:
>
> https://docs.docker.com/machine/
> https://docs.docker.com/swarm/overview/
>
>
> If we do it as a configuration, then I guess we should use
> org.apache.taverna.configuration Configuration Manager
>
>
> https://taverna.incubator.apache.org/javadoc/taverna-osgi/org/apache/taverna/configuration/Configurable.html
>
> see for instance
>
> https://github.com/apache/incubator-taverna-engine/blob/master/taverna-database-configuration-impl/src/main/java/org/apache/taverna/configuration/database/impl/DatabaseConfigurationImpl.java
>
>
> then users could modify the equivalent of
>
> conf/docker.conf
>
> to specify those defaults.
>
>
>
> On 5 June 2016 at 05:52, Nadeesh Dilanga <na...@gmail.com> wrote:
> > Hi Stian,
> > New container for every execution means, we actually not start a
> container
> > with a given name, but automatically generated name as you mentioned and
> > just start container in every workflow node execution ? Can one workflow
> > have more than one Docker invocation steps ?
> >
> > Yes, now it really make sense why we need a JSON format for Docker run.
> And
> > that should be the Docker run JSON payload.
> >
> > Following is a simple format I tried to invoke a Docker run(start a
> > container).
> >
> > curl X POST H "ContentType: application/json" d '{
> >
> "Hostname":"","User":"","Memory":0,"MemorySwap":0,"AttachStdin":false,"AttachStdout":true,"Attachstderr":true,
> >
> "PortSpecs":null,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":["date"],"Image":"ubuntu",
> > "Tag":"latest","Volumes":{"/tmp":{}
> > },"WorkingDir":"","DisableNetwork":false,"ExposedPorts":{"22/tcp": {} }
> >
> > Here is an example from the api doc which has more additional elements to
> > pass, request [2] and response[3].
> >
> > And sure, let's start with dumping logs to stdout.
> >
> > So the question now is whether we are going ahead with a full [2] like
> > comprehensive request. And on top of that, Host name is a mandatory field
> > which we need input from the workflow interface. What do you think ?
> >
> >
> > [2]
> > POST /containers/create HTTP/1.1 Content-Type: application/json {
> "Hostname":
> > "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout":
> true,
> > "AttachStderr": true, "Tty": false, "OpenStdin": false, "StdinOnce":
> false,
> > "Env": [ "FOO=bar", "BAZ=quux" ], "Cmd": [ "date" ], "Entrypoint": "", "
> > Image": "ubuntu", "Labels": { "com.example.vendor": "Acme", "
> > com.example.license": "GPL", "com.example.version": "1.0" }, "Volumes":
> { "
> > /volumes/data": {} }, "WorkingDir": "", "NetworkDisabled": false, "
> > MacAddress": "12:34:56:78:9a:bc", "ExposedPorts": { "22/tcp": {} }, "
> > StopSignal": "SIGTERM", "HostConfig": { "Binds": ["/tmp:/tmp"], "Links":
> [
> > "redis3:redis"], "Memory": 0, "MemorySwap": 0, "MemoryReservation": 0, "
> > KernelMemory": 0, "CpuShares": 512, "CpuPeriod": 100000, "CpuQuota":
> 50000,
> > "CpusetCpus": "0,1", "CpusetMems": "0,1", "BlkioWeight": 300, "
> > BlkioWeightDevice": [{}], "BlkioDeviceReadBps": [{}],
> "BlkioDeviceReadIOps":
> > [{}], "BlkioDeviceWriteBps": [{}], "BlkioDeviceWriteIOps": [{}], "
> > MemorySwappiness": 60, "OomKillDisable": false, "OomScoreAdj": 500, "
> > PidsLimit": -1, "PortBindings": { "22/tcp": [{ "HostPort": "11022" }] },
> "
> > PublishAllPorts": false, "Privileged": false, "ReadonlyRootfs": false,
> "Dns":
> > ["8.8.8.8"], "DnsOptions": [""], "DnsSearch": [""], "ExtraHosts": null, "
> > VolumesFrom": ["parent", "other:ro"], "CapAdd": ["NET_ADMIN"],
> "CapDrop": [
> > "MKNOD"], "GroupAdd": ["newgroup"], "RestartPolicy": { "Name": "", "
> > MaximumRetryCount": 0 }, "NetworkMode": "bridge", "Devices": [],
> "Ulimits":
> > [{}], "LogConfig": { "Type": "json-file", "Config": {} }, "SecurityOpt":
> > [], "CgroupParent": "", "VolumeDriver": "", "ShmSize": 67108864 }, "
> > NetworkingConfig": { "EndpointsConfig": { "isolated_nw" : {
> "IPAMConfig": {
> > "IPv4Address":"172.20.30.33", "IPv6Address":"2001:db8:abcd::3033" },
> "Links
> > ":["container_1", "container_2"], "Aliases":["server_x", "server_y"] } }
> }
> >
> >
> > [3] And response comes as
> >
> > HTTP/1.1 201 Created Content-Type: application/json {
> "Id":"e90e34656806",
> > "Warnings":[] }
> >
> >
> >
> >
> > On Sat, Jun 4, 2016 at 6:39 AM, Stian Soiland-Reyes <st...@apache.org>
> > wrote:
> >
> >> Hi!
> >>
> >> I think we can assume that docker images are in the registry, although I
> >> know it's possible to specify alternate registries as well, which is
> useful
> >> for commercial entities. In could even be possible to store a custom
> docker
> >> image within the Taverna wfbundle zip, although it would make it
> ginourmous
> >> ;) ,-- so let's start by assuming a public image in the Docket hub,
> >> probably versioned.
> >>
> >> Secondly, no, I don't think we can assume the container exists, as
> >> workflows can be shared. A good motivator for the functionality you are
> >> building is that a Taverna workflow that relies on command line tools
> can
> >> be shared with others who don't have those tools already installed (in
> >> expected location, path etc).
> >>
> >> I think for many use cases the container Name is not important and can
> be
> >> new for every executions, e.g. let Docker assign it's usual Jolly Badger
> >> style names.
> >>
> >> What is the Docker API for setting up a container? Do you need to pull
> the
> >> images first? It would be cool if we can pull at the beginning of a
> >> workflow run (e.g. asynchronously when an Activity is configured) so
> that
> >> does not need to delay execution at a later step.
> >>
> >> We will probably need a volume mount of a temporary directory, so that
> >> input and output files can be provided to the command, but if you
> prefer it
> >> might be easier to start with stdin and stdout support; similar to the
> Tool
> >> Activity.
> >> On 4 Jun 2016 5:10 a.m., "Nadeesh Dilanga" <na...@gmail.com>
> wrote:
> >>
> >> > Hi all,
> >> > I am starting this thread to discuss and finalize the docker commands
> we
> >> > need to expose for client side(Taverna).
> >> >
> >> > Latest stable docker remote API is version 1.23[1]. And it has several
> >> APIs
> >> > that can be useful.
> >> >
> >> > The original JIRA [2] mentioned about the JSON format to a docker
> run. I
> >> > hope it meant about the docker config.json ?
> >> >
> >> > Because, given we use remote APIs, I would like to know what are the
> >> > expectations are ?
> >> >
> >> > 1. Do we assume that Images are created and published to the registry.
> >> > 2. Do we assume that docker container is created
> >> >
> >> > Given #1 and #2 done, then we are talking about starting the
> >> > container(~docker run). If that is the case, when we use remote APIs
> we
> >> > only need following, and no need of a JSON:
> >> >
> >> > Request: POST /containers/(id or name)/start
> >> > Response: HTTP/1.1 204 No Content
> >> >
> >> > There are other responses too:
> >> >
> >> > Status Codes:
> >> >
> >> > - *204* – no error
> >> > - *304* – container already started
> >> > - *404* – no such container
> >> > - *500* – server error
> >> >
> >> >
> >> >
> >> > [1] -
> >> > https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/
> >> >
> >>
>
>
>
> --
> Stian Soiland-Reyes
> Apache Taverna (incubating), Apache Commons
> http://orcid.org/0000-0001-9842-9718
>
Re: Finalize Docker Invoke JSON format
Posted by Stian Soiland-Reyes <st...@apache.org>.
Perhaps we need a configuration/preferences per Workbench/workflow
about where to run the docker images. It's going to be different per
user.. e.g. if you use Docker on a Mac then it will be some local IP
address to the virtual machine - but then it's probably going to be
the same for each workflow, or for each step in the same workflow.
So do you think we need it to be part of the Activity configuration?
Perhaps unnecessary flexibility..
I don't think we need to aim for a Docker Machine-type or Docker
Swarm-type scenario, although that would be cool to keep in mind to
enable cloud-based workflows:
https://docs.docker.com/machine/
https://docs.docker.com/swarm/overview/
If we do it as a configuration, then I guess we should use
org.apache.taverna.configuration Configuration Manager
https://taverna.incubator.apache.org/javadoc/taverna-osgi/org/apache/taverna/configuration/Configurable.html
see for instance
https://github.com/apache/incubator-taverna-engine/blob/master/taverna-database-configuration-impl/src/main/java/org/apache/taverna/configuration/database/impl/DatabaseConfigurationImpl.java
then users could modify the equivalent of
conf/docker.conf
to specify those defaults.
On 5 June 2016 at 05:52, Nadeesh Dilanga <na...@gmail.com> wrote:
> Hi Stian,
> New container for every execution means, we actually not start a container
> with a given name, but automatically generated name as you mentioned and
> just start container in every workflow node execution ? Can one workflow
> have more than one Docker invocation steps ?
>
> Yes, now it really make sense why we need a JSON format for Docker run. And
> that should be the Docker run JSON payload.
>
> Following is a simple format I tried to invoke a Docker run(start a
> container).
>
> curl X POST H "ContentType: application/json" d '{
> "Hostname":"","User":"","Memory":0,"MemorySwap":0,"AttachStdin":false,"AttachStdout":true,"Attachstderr":true,
> "PortSpecs":null,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":["date"],"Image":"ubuntu",
> "Tag":"latest","Volumes":{"/tmp":{}
> },"WorkingDir":"","DisableNetwork":false,"ExposedPorts":{"22/tcp": {} }
>
> Here is an example from the api doc which has more additional elements to
> pass, request [2] and response[3].
>
> And sure, let's start with dumping logs to stdout.
>
> So the question now is whether we are going ahead with a full [2] like
> comprehensive request. And on top of that, Host name is a mandatory field
> which we need input from the workflow interface. What do you think ?
>
>
> [2]
> POST /containers/create HTTP/1.1 Content-Type: application/json { "Hostname":
> "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": true,
> "AttachStderr": true, "Tty": false, "OpenStdin": false, "StdinOnce": false,
> "Env": [ "FOO=bar", "BAZ=quux" ], "Cmd": [ "date" ], "Entrypoint": "", "
> Image": "ubuntu", "Labels": { "com.example.vendor": "Acme", "
> com.example.license": "GPL", "com.example.version": "1.0" }, "Volumes": { "
> /volumes/data": {} }, "WorkingDir": "", "NetworkDisabled": false, "
> MacAddress": "12:34:56:78:9a:bc", "ExposedPorts": { "22/tcp": {} }, "
> StopSignal": "SIGTERM", "HostConfig": { "Binds": ["/tmp:/tmp"], "Links": [
> "redis3:redis"], "Memory": 0, "MemorySwap": 0, "MemoryReservation": 0, "
> KernelMemory": 0, "CpuShares": 512, "CpuPeriod": 100000, "CpuQuota": 50000,
> "CpusetCpus": "0,1", "CpusetMems": "0,1", "BlkioWeight": 300, "
> BlkioWeightDevice": [{}], "BlkioDeviceReadBps": [{}], "BlkioDeviceReadIOps":
> [{}], "BlkioDeviceWriteBps": [{}], "BlkioDeviceWriteIOps": [{}], "
> MemorySwappiness": 60, "OomKillDisable": false, "OomScoreAdj": 500, "
> PidsLimit": -1, "PortBindings": { "22/tcp": [{ "HostPort": "11022" }] }, "
> PublishAllPorts": false, "Privileged": false, "ReadonlyRootfs": false, "Dns":
> ["8.8.8.8"], "DnsOptions": [""], "DnsSearch": [""], "ExtraHosts": null, "
> VolumesFrom": ["parent", "other:ro"], "CapAdd": ["NET_ADMIN"], "CapDrop": [
> "MKNOD"], "GroupAdd": ["newgroup"], "RestartPolicy": { "Name": "", "
> MaximumRetryCount": 0 }, "NetworkMode": "bridge", "Devices": [], "Ulimits":
> [{}], "LogConfig": { "Type": "json-file", "Config": {} }, "SecurityOpt":
> [], "CgroupParent": "", "VolumeDriver": "", "ShmSize": 67108864 }, "
> NetworkingConfig": { "EndpointsConfig": { "isolated_nw" : { "IPAMConfig": {
> "IPv4Address":"172.20.30.33", "IPv6Address":"2001:db8:abcd::3033" }, "Links
> ":["container_1", "container_2"], "Aliases":["server_x", "server_y"] } } }
>
>
> [3] And response comes as
>
> HTTP/1.1 201 Created Content-Type: application/json { "Id":"e90e34656806",
> "Warnings":[] }
>
>
>
>
> On Sat, Jun 4, 2016 at 6:39 AM, Stian Soiland-Reyes <st...@apache.org>
> wrote:
>
>> Hi!
>>
>> I think we can assume that docker images are in the registry, although I
>> know it's possible to specify alternate registries as well, which is useful
>> for commercial entities. In could even be possible to store a custom docker
>> image within the Taverna wfbundle zip, although it would make it ginourmous
>> ;) ,-- so let's start by assuming a public image in the Docket hub,
>> probably versioned.
>>
>> Secondly, no, I don't think we can assume the container exists, as
>> workflows can be shared. A good motivator for the functionality you are
>> building is that a Taverna workflow that relies on command line tools can
>> be shared with others who don't have those tools already installed (in
>> expected location, path etc).
>>
>> I think for many use cases the container Name is not important and can be
>> new for every executions, e.g. let Docker assign it's usual Jolly Badger
>> style names.
>>
>> What is the Docker API for setting up a container? Do you need to pull the
>> images first? It would be cool if we can pull at the beginning of a
>> workflow run (e.g. asynchronously when an Activity is configured) so that
>> does not need to delay execution at a later step.
>>
>> We will probably need a volume mount of a temporary directory, so that
>> input and output files can be provided to the command, but if you prefer it
>> might be easier to start with stdin and stdout support; similar to the Tool
>> Activity.
>> On 4 Jun 2016 5:10 a.m., "Nadeesh Dilanga" <na...@gmail.com> wrote:
>>
>> > Hi all,
>> > I am starting this thread to discuss and finalize the docker commands we
>> > need to expose for client side(Taverna).
>> >
>> > Latest stable docker remote API is version 1.23[1]. And it has several
>> APIs
>> > that can be useful.
>> >
>> > The original JIRA [2] mentioned about the JSON format to a docker run. I
>> > hope it meant about the docker config.json ?
>> >
>> > Because, given we use remote APIs, I would like to know what are the
>> > expectations are ?
>> >
>> > 1. Do we assume that Images are created and published to the registry.
>> > 2. Do we assume that docker container is created
>> >
>> > Given #1 and #2 done, then we are talking about starting the
>> > container(~docker run). If that is the case, when we use remote APIs we
>> > only need following, and no need of a JSON:
>> >
>> > Request: POST /containers/(id or name)/start
>> > Response: HTTP/1.1 204 No Content
>> >
>> > There are other responses too:
>> >
>> > Status Codes:
>> >
>> > - *204* – no error
>> > - *304* – container already started
>> > - *404* – no such container
>> > - *500* – server error
>> >
>> >
>> >
>> > [1] -
>> > https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/
>> >
>>
--
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons
http://orcid.org/0000-0001-9842-9718
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi Stian,
New container for every execution means, we actually not start a container
with a given name, but automatically generated name as you mentioned and
just start container in every workflow node execution ? Can one workflow
have more than one Docker invocation steps ?
Yes, now it really make sense why we need a JSON format for Docker run. And
that should be the Docker run JSON payload.
Following is a simple format I tried to invoke a Docker run(start a
container).
curl X POST H "ContentType: application/json" d '{
"Hostname":"","User":"","Memory":0,"MemorySwap":0,"AttachStdin":false,"AttachStdout":true,"Attachstderr":true,
"PortSpecs":null,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":["date"],"Image":"ubuntu",
"Tag":"latest","Volumes":{"/tmp":{}
},"WorkingDir":"","DisableNetwork":false,"ExposedPorts":{"22/tcp": {} }
Here is an example from the api doc which has more additional elements to
pass, request [2] and response[3].
And sure, let's start with dumping logs to stdout.
So the question now is whether we are going ahead with a full [2] like
comprehensive request. And on top of that, Host name is a mandatory field
which we need input from the workflow interface. What do you think ?
[2]
POST /containers/create HTTP/1.1 Content-Type: application/json { "Hostname":
"", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": true,
"AttachStderr": true, "Tty": false, "OpenStdin": false, "StdinOnce": false,
"Env": [ "FOO=bar", "BAZ=quux" ], "Cmd": [ "date" ], "Entrypoint": "", "
Image": "ubuntu", "Labels": { "com.example.vendor": "Acme", "
com.example.license": "GPL", "com.example.version": "1.0" }, "Volumes": { "
/volumes/data": {} }, "WorkingDir": "", "NetworkDisabled": false, "
MacAddress": "12:34:56:78:9a:bc", "ExposedPorts": { "22/tcp": {} }, "
StopSignal": "SIGTERM", "HostConfig": { "Binds": ["/tmp:/tmp"], "Links": [
"redis3:redis"], "Memory": 0, "MemorySwap": 0, "MemoryReservation": 0, "
KernelMemory": 0, "CpuShares": 512, "CpuPeriod": 100000, "CpuQuota": 50000,
"CpusetCpus": "0,1", "CpusetMems": "0,1", "BlkioWeight": 300, "
BlkioWeightDevice": [{}], "BlkioDeviceReadBps": [{}], "BlkioDeviceReadIOps":
[{}], "BlkioDeviceWriteBps": [{}], "BlkioDeviceWriteIOps": [{}], "
MemorySwappiness": 60, "OomKillDisable": false, "OomScoreAdj": 500, "
PidsLimit": -1, "PortBindings": { "22/tcp": [{ "HostPort": "11022" }] }, "
PublishAllPorts": false, "Privileged": false, "ReadonlyRootfs": false, "Dns":
["8.8.8.8"], "DnsOptions": [""], "DnsSearch": [""], "ExtraHosts": null, "
VolumesFrom": ["parent", "other:ro"], "CapAdd": ["NET_ADMIN"], "CapDrop": [
"MKNOD"], "GroupAdd": ["newgroup"], "RestartPolicy": { "Name": "", "
MaximumRetryCount": 0 }, "NetworkMode": "bridge", "Devices": [], "Ulimits":
[{}], "LogConfig": { "Type": "json-file", "Config": {} }, "SecurityOpt":
[], "CgroupParent": "", "VolumeDriver": "", "ShmSize": 67108864 }, "
NetworkingConfig": { "EndpointsConfig": { "isolated_nw" : { "IPAMConfig": {
"IPv4Address":"172.20.30.33", "IPv6Address":"2001:db8:abcd::3033" }, "Links
":["container_1", "container_2"], "Aliases":["server_x", "server_y"] } } }
[3] And response comes as
HTTP/1.1 201 Created Content-Type: application/json { "Id":"e90e34656806",
"Warnings":[] }
On Sat, Jun 4, 2016 at 6:39 AM, Stian Soiland-Reyes <st...@apache.org>
wrote:
> Hi!
>
> I think we can assume that docker images are in the registry, although I
> know it's possible to specify alternate registries as well, which is useful
> for commercial entities. In could even be possible to store a custom docker
> image within the Taverna wfbundle zip, although it would make it ginourmous
> ;) ,-- so let's start by assuming a public image in the Docket hub,
> probably versioned.
>
> Secondly, no, I don't think we can assume the container exists, as
> workflows can be shared. A good motivator for the functionality you are
> building is that a Taverna workflow that relies on command line tools can
> be shared with others who don't have those tools already installed (in
> expected location, path etc).
>
> I think for many use cases the container Name is not important and can be
> new for every executions, e.g. let Docker assign it's usual Jolly Badger
> style names.
>
> What is the Docker API for setting up a container? Do you need to pull the
> images first? It would be cool if we can pull at the beginning of a
> workflow run (e.g. asynchronously when an Activity is configured) so that
> does not need to delay execution at a later step.
>
> We will probably need a volume mount of a temporary directory, so that
> input and output files can be provided to the command, but if you prefer it
> might be easier to start with stdin and stdout support; similar to the Tool
> Activity.
> On 4 Jun 2016 5:10 a.m., "Nadeesh Dilanga" <na...@gmail.com> wrote:
>
> > Hi all,
> > I am starting this thread to discuss and finalize the docker commands we
> > need to expose for client side(Taverna).
> >
> > Latest stable docker remote API is version 1.23[1]. And it has several
> APIs
> > that can be useful.
> >
> > The original JIRA [2] mentioned about the JSON format to a docker run. I
> > hope it meant about the docker config.json ?
> >
> > Because, given we use remote APIs, I would like to know what are the
> > expectations are ?
> >
> > 1. Do we assume that Images are created and published to the registry.
> > 2. Do we assume that docker container is created
> >
> > Given #1 and #2 done, then we are talking about starting the
> > container(~docker run). If that is the case, when we use remote APIs we
> > only need following, and no need of a JSON:
> >
> > Request: POST /containers/(id or name)/start
> > Response: HTTP/1.1 204 No Content
> >
> > There are other responses too:
> >
> > Status Codes:
> >
> > - *204* – no error
> > - *304* – container already started
> > - *404* – no such container
> > - *500* – server error
> >
> >
> >
> > [1] -
> > https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/
> >
>
Re: Finalize Docker Invoke JSON format
Posted by Stian Soiland-Reyes <st...@apache.org>.
Hi!
I think we can assume that docker images are in the registry, although I
know it's possible to specify alternate registries as well, which is useful
for commercial entities. In could even be possible to store a custom docker
image within the Taverna wfbundle zip, although it would make it ginourmous
;) ,-- so let's start by assuming a public image in the Docket hub,
probably versioned.
Secondly, no, I don't think we can assume the container exists, as
workflows can be shared. A good motivator for the functionality you are
building is that a Taverna workflow that relies on command line tools can
be shared with others who don't have those tools already installed (in
expected location, path etc).
I think for many use cases the container Name is not important and can be
new for every executions, e.g. let Docker assign it's usual Jolly Badger
style names.
What is the Docker API for setting up a container? Do you need to pull the
images first? It would be cool if we can pull at the beginning of a
workflow run (e.g. asynchronously when an Activity is configured) so that
does not need to delay execution at a later step.
We will probably need a volume mount of a temporary directory, so that
input and output files can be provided to the command, but if you prefer it
might be easier to start with stdin and stdout support; similar to the Tool
Activity.
On 4 Jun 2016 5:10 a.m., "Nadeesh Dilanga" <na...@gmail.com> wrote:
> Hi all,
> I am starting this thread to discuss and finalize the docker commands we
> need to expose for client side(Taverna).
>
> Latest stable docker remote API is version 1.23[1]. And it has several APIs
> that can be useful.
>
> The original JIRA [2] mentioned about the JSON format to a docker run. I
> hope it meant about the docker config.json ?
>
> Because, given we use remote APIs, I would like to know what are the
> expectations are ?
>
> 1. Do we assume that Images are created and published to the registry.
> 2. Do we assume that docker container is created
>
> Given #1 and #2 done, then we are talking about starting the
> container(~docker run). If that is the case, when we use remote APIs we
> only need following, and no need of a JSON:
>
> Request: POST /containers/(id or name)/start
> Response: HTTP/1.1 204 No Content
>
> There are other responses too:
>
> Status Codes:
>
> - *204* – no error
> - *304* – container already started
> - *404* – no such container
> - *500* – server error
>
>
>
> [1] -
> https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/
>
Re: Finalize Docker Invoke JSON format
Posted by Stian Soiland-Reyes <st...@apache.org>.
Is it possible/easy to submit stdin input via the REST API?
The API discusses if "One client attaches to STDIN" under the "Attach
to a container" section.
https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/#attach-to-a-container
this uses HTTP 1.1 upgrade to a pure TCP connection - this would put
restrictions on how you can do the HTTP connection, or is that easy to
do say with Apache HTTP Components?
Perhaps look at existing Docker APIs for Java?
https://github.com/docker-java/docker-java (Apache license)
https://github.com/spotify/docker-client (Apache license)
Sending from Taverna should be easy as you can ask for an input stream
from every kind of reference, and just forward the bytes straight to
that attached stream, e.g. with CommonIO.copy(InputStream,
OutputStream).
As a good simple test for stdin/stdout, perhaps "wc" in the busybox
image to count words - e.g. replicating something like:
stain@biggiebuntu:~/src/taverna$ echo "Hello there man" | docker run
-i busybox wc
1 3 16
On 8 June 2016 at 01:52, Nadeesh Dilanga <na...@gmail.com> wrote:
> Hi Alan,
> Can you please elaborate what you mean by " It is not sufficient to just
> start the container if it expects input data to be sent to stdin, or to be
> in specific locations." ?
>
> When starting a container using remote API, the only response we get is a
> HTTP status code along with a message body with container id which
> indicates the container started successfully. I assume you meant more on IF
> the application in the container expects input data to be applied and sent
> to stdin, and for such cases how we are going to cater that requirement ?
>
>
>
> On Mon, Jun 6, 2016 at 4:33 AM, Alan Williams <al...@googlemail.com>
> wrote:
>
>> On 04-Jun-16 05:10, Nadeesh Dilanga wrote:
>>
>>> Hi all,
>>> I am starting this thread to discuss and finalize the docker commands we
>>> need to expose for client side(Taverna).
>>>
>>> Latest stable docker remote API is version 1.23[1]. And it has several
>>> APIs
>>> that can be useful.
>>>
>>
>> Yes. Where do you intend to run the docker containers? That would be
>> similar to how you can specify where to run tool services - although I
>> dislike how that is done.
>>
>> The original JIRA [2] mentioned about the JSON format to a docker run. I
>>> hope it meant about the docker config.json ?
>>>
>>
>> That would be a lot of configuration. I am not sure that it is sensible to
>> have it all specified in the workflow.
>>
>> Because, given we use remote APIs, I would like to know what are the
>>> expectations are ?
>>>
>>> 1. Do we assume that Images are created and published to the registry.
>>>
>>
>> Yes, certainly for running you will know the registry and the image.
>>
>> 2. Do we assume that docker container is created
>>>
>>
>> I don't think you can, as that would mean there are steps needed to be
>> done before the workflow can be run.
>>
>> Given #1 and #2 done, then we are talking about starting the
>>> container(~docker run). If that is the case, when we use remote APIs we
>>> only need following, and no need of a JSON:
>>>
>>> Request: POST /containers/(id or name)/start
>>> Response: HTTP/1.1 204 No Content
>>>
>>> There are other responses too:
>>>
>>> Status Codes:
>>>
>>> - *204* – no error
>>> - *304* – container already started
>>> - *404* – no such container
>>> - *500* – server error
>>>
>>
>> You will need to look at how the input data is read and the results
>> returned. It is not sufficient to just start the container if it expects
>> input data to be sent to stdin, or to be in specific locations.
>>
>> [1] -
>>> https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/
>>>
>>
>> Alan
>>
>>
>>
--
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons
http://orcid.org/0000-0001-9842-9718
Re: Finalize Docker Invoke JSON format
Posted by Nadeesh Dilanga <na...@gmail.com>.
Hi Alan,
Can you please elaborate what you mean by " It is not sufficient to just
start the container if it expects input data to be sent to stdin, or to be
in specific locations." ?
When starting a container using remote API, the only response we get is a
HTTP status code along with a message body with container id which
indicates the container started successfully. I assume you meant more on IF
the application in the container expects input data to be applied and sent
to stdin, and for such cases how we are going to cater that requirement ?
On Mon, Jun 6, 2016 at 4:33 AM, Alan Williams <al...@googlemail.com>
wrote:
> On 04-Jun-16 05:10, Nadeesh Dilanga wrote:
>
>> Hi all,
>> I am starting this thread to discuss and finalize the docker commands we
>> need to expose for client side(Taverna).
>>
>> Latest stable docker remote API is version 1.23[1]. And it has several
>> APIs
>> that can be useful.
>>
>
> Yes. Where do you intend to run the docker containers? That would be
> similar to how you can specify where to run tool services - although I
> dislike how that is done.
>
> The original JIRA [2] mentioned about the JSON format to a docker run. I
>> hope it meant about the docker config.json ?
>>
>
> That would be a lot of configuration. I am not sure that it is sensible to
> have it all specified in the workflow.
>
> Because, given we use remote APIs, I would like to know what are the
>> expectations are ?
>>
>> 1. Do we assume that Images are created and published to the registry.
>>
>
> Yes, certainly for running you will know the registry and the image.
>
> 2. Do we assume that docker container is created
>>
>
> I don't think you can, as that would mean there are steps needed to be
> done before the workflow can be run.
>
> Given #1 and #2 done, then we are talking about starting the
>> container(~docker run). If that is the case, when we use remote APIs we
>> only need following, and no need of a JSON:
>>
>> Request: POST /containers/(id or name)/start
>> Response: HTTP/1.1 204 No Content
>>
>> There are other responses too:
>>
>> Status Codes:
>>
>> - *204* – no error
>> - *304* – container already started
>> - *404* – no such container
>> - *500* – server error
>>
>
> You will need to look at how the input data is read and the results
> returned. It is not sufficient to just start the container if it expects
> input data to be sent to stdin, or to be in specific locations.
>
> [1] -
>> https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/
>>
>
> Alan
>
>
>
Re: Finalize Docker Invoke JSON format
Posted by Alan Williams <al...@googlemail.com>.
On 04-Jun-16 05:10, Nadeesh Dilanga wrote:
> Hi all,
> I am starting this thread to discuss and finalize the docker commands we
> need to expose for client side(Taverna).
>
> Latest stable docker remote API is version 1.23[1]. And it has several APIs
> that can be useful.
Yes. Where do you intend to run the docker containers? That would be
similar to how you can specify where to run tool services - although I
dislike how that is done.
> The original JIRA [2] mentioned about the JSON format to a docker run. I
> hope it meant about the docker config.json ?
That would be a lot of configuration. I am not sure that it is sensible
to have it all specified in the workflow.
> Because, given we use remote APIs, I would like to know what are the
> expectations are ?
>
> 1. Do we assume that Images are created and published to the registry.
Yes, certainly for running you will know the registry and the image.
> 2. Do we assume that docker container is created
I don't think you can, as that would mean there are steps needed to be
done before the workflow can be run.
> Given #1 and #2 done, then we are talking about starting the
> container(~docker run). If that is the case, when we use remote APIs we
> only need following, and no need of a JSON:
>
> Request: POST /containers/(id or name)/start
> Response: HTTP/1.1 204 No Content
>
> There are other responses too:
>
> Status Codes:
>
> - *204* \u2013 no error
> - *304* \u2013 container already started
> - *404* \u2013 no such container
> - *500* \u2013 server error
You will need to look at how the input data is read and the results
returned. It is not sufficient to just start the container if it expects
input data to be sent to stdin, or to be in specific locations.
> [1] - https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/
Alan