You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Tom Barber <to...@spicule.co.uk> on 2019/05/14 14:12:43 UTC
[GSOC] OODT docker build meeting summary
Just a quick recap from our call with Yasith earlier today:
Had a good chat with Yasith about how RADIX works and how people build out
OODT applications from RADIX (
https://cwiki.apache.org/confluence/display/OODT/RADiX+Powered+By+OODT)
General Docker build plan is as follows:
* Users can deploy a standard RADIX build and optionally choose the Docker
profile at build time to build docker images from it
* Docker will be optional not standard
* RADIX will allow users to define the image tag names for easy tagging and
pushing to docker hub or similar
* In the GSOC version we will focus on single deployments of each component
not clustered up using ZK or similar we will stick that in the TODO list
Once built the end target is Kubernetes, both MiniKube and full Kubernetes.
Docker Compose may be used for interim testing and validation.
A few deploy methods exist for validation:
Standard deploy yams for K8S
Helm Chart
Or via a custom Kubernetes Operator similar to ArangoDB (
https://github.com/arangodb/kube-arangodb)
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: [GSOC] OODT docker build meeting summary
Posted by Imesha Sudasingha <im...@apache.org>.
On today's meeting, it was discussed to;
1. Write a user guide and overall blog post describing how to build and use
the docker images
2. Yasith will also work on a kubernetes services file
3. The OPSUI container is unable to contact resource manager. This issue
requires to be investigates and fixed.
Thanks,
Imesha
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Fri, 26 Jul 2019 at 11:59, Jayawardana, Yasith <ya...@cs.odu.edu> wrote:
> Hi Tom and Imesha,
>
> A quick follow up question: as we are soon going to have the dockerfile PR
> finalized, I believe what’s next is the deployment configuration for it. Is
> integrating the distributed configuration management in the scope for this
> project, or should we focus on creating the docker-compose/kubernetes
> service configurations as standalone (using the predefined configurations)
> for now? IMHO, if we get the standalone configuration completed and
> documented, then implementing the distributed configuration would be an
> extension of it, right? Or is it a completely goal effort to achieve?
>
> Best Regards,
> Yasith Jayawardana
>
> ------------------------------------------------
> Yasith Jayawardana
> PhD Student | Old Dominion University
> phone: +1 757 698 8978
>
> On Jul 26, 2019, at 2:20 AM, Jayawardana, Yasith <yasith@cs.odu.edu
> <ma...@cs.odu.edu>> wrote:
>
> Hi All,
>
> This is a quick update regarding the current status of the OODT Docker
> Deployment project, and what’s next.
>
> During the last two meetings with @Imesha, we decided to use docker named
> volumes to mount the /data directory, which contains the catalog and
> archive content. The other directories will use the default structure
> enforced by the RADiX build. However, if the user needs, they can mount
> directories on top of the existing ones to specify new policies, etc. But
> this will be outside the default dockerfiles and handled at Docker Compose
> level. Also, I will be working on the comments made by @Imesha on the PR
> and fix things along the way. I am also working on a new Documentation that
> discusses about the docker build process, and add it to the Confluence wiki
> in the upcoming weeks. Also I’ll be working on quick start guide using the
> docker build process, which will not be as extensive as the documentation,
> but serves as a barebones guide to get a user up and running.
>
> Best Regards,
> Yasith Jayawardana
>
> ------------------------------------------------
> Yasith Jayawardana
> PhD Student | Old Dominion University
> phone: +1 757 698 8978
>
> On Jul 24, 2019, at 12:50 AM, Imesha Sudasingha <imesha@apache.org<mailto:
> imesha@apache.org><ma...@apache.org>> wrote:
>
> Hi Yasith,
>
> Thanks for the PRs. I have added some comments to them. Could you check?
>
> On Wed, 17 Jul 2019 at 13:31, Jayawardana, Yasith <yasith@cs.odu.edu
> <ma...@cs.odu.edu>> wrote:
>
> Hi all,
>
> I created two PRs based on the discussion with @Imesha and @Tom over the
> last weeks. The two PRs are as follows:
>
> *
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Foodt%2Fpull%2F100&data=02%7C01%7Cyasith%40cs.odu.edu%7Cec02797c980d42b08a1108d7119160b2%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636997188429111104&sdata=KnFaAMuAoaqiYPJRObje%2FOT6aLrbmm4s14YWZ2nWt74%3D&reserved=0
> - This contains the Dockerfiles
> and modifications to existing files to accommodate docker image creation
> workflow in radix. I moved this workflow into a separate build profile
> named “docker” so that it won’t break existing workflows.
>
> *
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Foodt%2Fpull%2F99&data=02%7C01%7Cyasith%40cs.odu.edu%7Cec02797c980d42b08a1108d7119160b2%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636997188429111104&sdata=CCfyTy9%2B6uD3NjAVztK8a8sA%2FgbQMr53ML4WKe%2BSLcQ%3D&reserved=0
> - Updated maven-assembly-plugin
> references in the radix archetype from 2.2-beta to 2.6. This was due to
> encountering errors when extracting the tar on certain distros (for my case
> it was alpine linux) After updating the plugin version the error was
> resolved, so thought it was worth a PR.
>
> As it was suggested, both PRs are aimed at “development” branch and not
> the “master” branch. Appreciate if you all could take a look and review it,
> and any suggestions are welcome!
>
> Best Regards,
> Yasith Jayawardana
>
> ------------------------------------------------
> Yasith Jayawardana
> PhD Student | Old Dominion University
> phone: +1 757 698 8978
>
> On Jul 4, 2019, at 2:20 PM, Imesha Sudasingha <imesha@apache.org<mailto:
> imesha@apache.org><ma...@apache.org><mailto:
> imesha@apache.org<ma...@apache.org>>>
> wrote:
>
> Hi Yasith,
>
> Thanks for the update! No worries about recent quietness. Let's focus on
> getting file manager container to interact with solr as the next step and
> look at what volumes require to be mounted. Once that is done, we can see
> how we can accommodate your code within OODT.
>
> Thanks,
> Imesha
>
>
>
>
>
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: [GSOC] OODT docker build meeting summary
Posted by "Jayawardana, Yasith" <ya...@cs.odu.edu>.
Hi Tom and Imesha,
A quick follow up question: as we are soon going to have the dockerfile PR finalized, I believe what’s next is the deployment configuration for it. Is integrating the distributed configuration management in the scope for this project, or should we focus on creating the docker-compose/kubernetes service configurations as standalone (using the predefined configurations) for now? IMHO, if we get the standalone configuration completed and documented, then implementing the distributed configuration would be an extension of it, right? Or is it a completely goal effort to achieve?
Best Regards,
Yasith Jayawardana
------------------------------------------------
Yasith Jayawardana
PhD Student | Old Dominion University
phone: +1 757 698 8978
On Jul 26, 2019, at 2:20 AM, Jayawardana, Yasith <ya...@cs.odu.edu>> wrote:
Hi All,
This is a quick update regarding the current status of the OODT Docker Deployment project, and what’s next.
During the last two meetings with @Imesha, we decided to use docker named volumes to mount the /data directory, which contains the catalog and archive content. The other directories will use the default structure enforced by the RADiX build. However, if the user needs, they can mount directories on top of the existing ones to specify new policies, etc. But this will be outside the default dockerfiles and handled at Docker Compose level. Also, I will be working on the comments made by @Imesha on the PR and fix things along the way. I am also working on a new Documentation that discusses about the docker build process, and add it to the Confluence wiki in the upcoming weeks. Also I’ll be working on quick start guide using the docker build process, which will not be as extensive as the documentation, but serves as a barebones guide to get a user up and running.
Best Regards,
Yasith Jayawardana
------------------------------------------------
Yasith Jayawardana
PhD Student | Old Dominion University
phone: +1 757 698 8978
On Jul 24, 2019, at 12:50 AM, Imesha Sudasingha <im...@apache.org>> wrote:
Hi Yasith,
Thanks for the PRs. I have added some comments to them. Could you check?
On Wed, 17 Jul 2019 at 13:31, Jayawardana, Yasith <ya...@cs.odu.edu>> wrote:
Hi all,
I created two PRs based on the discussion with @Imesha and @Tom over the
last weeks. The two PRs are as follows:
* https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Foodt%2Fpull%2F100&data=02%7C01%7Cyasith%40cs.odu.edu%7Cec02797c980d42b08a1108d7119160b2%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636997188429111104&sdata=KnFaAMuAoaqiYPJRObje%2FOT6aLrbmm4s14YWZ2nWt74%3D&reserved=0 - This contains the Dockerfiles
and modifications to existing files to accommodate docker image creation
workflow in radix. I moved this workflow into a separate build profile
named “docker” so that it won’t break existing workflows.
* https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Foodt%2Fpull%2F99&data=02%7C01%7Cyasith%40cs.odu.edu%7Cec02797c980d42b08a1108d7119160b2%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636997188429111104&sdata=CCfyTy9%2B6uD3NjAVztK8a8sA%2FgbQMr53ML4WKe%2BSLcQ%3D&reserved=0 - Updated maven-assembly-plugin
references in the radix archetype from 2.2-beta to 2.6. This was due to
encountering errors when extracting the tar on certain distros (for my case
it was alpine linux) After updating the plugin version the error was
resolved, so thought it was worth a PR.
As it was suggested, both PRs are aimed at “development” branch and not
the “master” branch. Appreciate if you all could take a look and review it,
and any suggestions are welcome!
Best Regards,
Yasith Jayawardana
------------------------------------------------
Yasith Jayawardana
PhD Student | Old Dominion University
phone: +1 757 698 8978
On Jul 4, 2019, at 2:20 PM, Imesha Sudasingha <im...@apache.org><mailto:
imesha@apache.org<ma...@apache.org>>> wrote:
Hi Yasith,
Thanks for the update! No worries about recent quietness. Let's focus on
getting file manager container to interact with solr as the next step and
look at what volumes require to be mounted. Once that is done, we can see
how we can accommodate your code within OODT.
Thanks,
Imesha
Re: [GSOC] OODT docker build meeting summary
Posted by "Jayawardana, Yasith" <ya...@cs.odu.edu>.
Hi All,
This is a quick update regarding the current status of the OODT Docker Deployment project, and what’s next.
During the last two meetings with @Imesha, we decided to use docker named volumes to mount the /data directory, which contains the catalog and archive content. The other directories will use the default structure enforced by the RADiX build. However, if the user needs, they can mount directories on top of the existing ones to specify new policies, etc. But this will be outside the default dockerfiles and handled at Docker Compose level. Also, I will be working on the comments made by @Imesha on the PR and fix things along the way. I am also working on a new Documentation that discusses about the docker build process, and add it to the Confluence wiki in the upcoming weeks. Also I’ll be working on quick start guide using the docker build process, which will not be as extensive as the documentation, but serves as a barebones guide to get a user up and running.
Best Regards,
Yasith Jayawardana
------------------------------------------------
Yasith Jayawardana
PhD Student | Old Dominion University
phone: +1 757 698 8978
On Jul 24, 2019, at 12:50 AM, Imesha Sudasingha <im...@apache.org>> wrote:
Hi Yasith,
Thanks for the PRs. I have added some comments to them. Could you check?
On Wed, 17 Jul 2019 at 13:31, Jayawardana, Yasith <ya...@cs.odu.edu>> wrote:
Hi all,
I created two PRs based on the discussion with @Imesha and @Tom over the
last weeks. The two PRs are as follows:
* https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Foodt%2Fpull%2F100&data=02%7C01%7Cyasith%40cs.odu.edu%7Cf055a6d3a6204bdaeab408d70ff29ca2%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C1%7C636995407008511885&sdata=4LV8nj7Og2Z0OJ46I2rqlLm2j%2FsWQ4wPqp3BkZ5%2BXyQ%3D&reserved=0 - This contains the Dockerfiles
and modifications to existing files to accommodate docker image creation
workflow in radix. I moved this workflow into a separate build profile
named “docker” so that it won’t break existing workflows.
* https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Foodt%2Fpull%2F99&data=02%7C01%7Cyasith%40cs.odu.edu%7Cf055a6d3a6204bdaeab408d70ff29ca2%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C1%7C636995407008511885&sdata=kbI9%2B31yZy9b%2BTyselFnV9aKe5O2Og1Xfw9%2B7tU0WBc%3D&reserved=0 - Updated maven-assembly-plugin
references in the radix archetype from 2.2-beta to 2.6. This was due to
encountering errors when extracting the tar on certain distros (for my case
it was alpine linux) After updating the plugin version the error was
resolved, so thought it was worth a PR.
As it was suggested, both PRs are aimed at “development” branch and not
the “master” branch. Appreciate if you all could take a look and review it,
and any suggestions are welcome!
Best Regards,
Yasith Jayawardana
------------------------------------------------
Yasith Jayawardana
PhD Student | Old Dominion University
phone: +1 757 698 8978
On Jul 4, 2019, at 2:20 PM, Imesha Sudasingha <im...@apache.org><mailto:
imesha@apache.org<ma...@apache.org>>> wrote:
Hi Yasith,
Thanks for the update! No worries about recent quietness. Let's focus on
getting file manager container to interact with solr as the next step and
look at what volumes require to be mounted. Once that is done, we can see
how we can accommodate your code within OODT.
Thanks,
Imesha
Re: [GSOC] OODT docker build meeting summary
Posted by Imesha Sudasingha <im...@apache.org>.
Hi Yasith,
Thanks for the PRs. I have added some comments to them. Could you check?
On Wed, 17 Jul 2019 at 13:31, Jayawardana, Yasith <ya...@cs.odu.edu> wrote:
> Hi all,
>
> I created two PRs based on the discussion with @Imesha and @Tom over the
> last weeks. The two PRs are as follows:
>
> * https://github.com/apache/oodt/pull/100 - This contains the Dockerfiles
> and modifications to existing files to accommodate docker image creation
> workflow in radix. I moved this workflow into a separate build profile
> named “docker” so that it won’t break existing workflows.
>
> * https://github.com/apache/oodt/pull/99 - Updated maven-assembly-plugin
> references in the radix archetype from 2.2-beta to 2.6. This was due to
> encountering errors when extracting the tar on certain distros (for my case
> it was alpine linux) After updating the plugin version the error was
> resolved, so thought it was worth a PR.
>
> As it was suggested, both PRs are aimed at “development” branch and not
> the “master” branch. Appreciate if you all could take a look and review it,
> and any suggestions are welcome!
>
> Best Regards,
> Yasith Jayawardana
>
> ------------------------------------------------
> Yasith Jayawardana
> PhD Student | Old Dominion University
> phone: +1 757 698 8978
>
> On Jul 4, 2019, at 2:20 PM, Imesha Sudasingha <imesha@apache.org<mailto:
> imesha@apache.org>> wrote:
>
> Hi Yasith,
>
> Thanks for the update! No worries about recent quietness. Let's focus on
> getting file manager container to interact with solr as the next step and
> look at what volumes require to be mounted. Once that is done, we can see
> how we can accommodate your code within OODT.
>
> Thanks,
> Imesha
>
>
Re: [GSOC] OODT docker build meeting summary
Posted by "Jayawardana, Yasith" <ya...@cs.odu.edu>.
Hi all,
I created two PRs based on the discussion with @Imesha and @Tom over the last weeks. The two PRs are as follows:
* https://github.com/apache/oodt/pull/100 - This contains the Dockerfiles and modifications to existing files to accommodate docker image creation workflow in radix. I moved this workflow into a separate build profile named “docker” so that it won’t break existing workflows.
* https://github.com/apache/oodt/pull/99 - Updated maven-assembly-plugin references in the radix archetype from 2.2-beta to 2.6. This was due to encountering errors when extracting the tar on certain distros (for my case it was alpine linux) After updating the plugin version the error was resolved, so thought it was worth a PR.
As it was suggested, both PRs are aimed at “development” branch and not the “master” branch. Appreciate if you all could take a look and review it, and any suggestions are welcome!
Best Regards,
Yasith Jayawardana
------------------------------------------------
Yasith Jayawardana
PhD Student | Old Dominion University
phone: +1 757 698 8978
On Jul 4, 2019, at 2:20 PM, Imesha Sudasingha <im...@apache.org>> wrote:
Hi Yasith,
Thanks for the update! No worries about recent quietness. Let's focus on
getting file manager container to interact with solr as the next step and
look at what volumes require to be mounted. Once that is done, we can see
how we can accommodate your code within OODT.
Thanks,
Imesha
Re: [GSOC] OODT docker build meeting summary
Posted by Imesha Sudasingha <im...@apache.org>.
Hi Yasith,
Thanks for the update! No worries about recent quietness. Let's focus on
getting file manager container to interact with solr as the next step and
look at what volumes require to be mounted. Once that is done, we can see
how we can accommodate your code within OODT.
Thanks,
Imesha
Re: [GSOC] OODT docker build meeting summary
Posted by "Jayawardana, Yasith" <ya...@cs.odu.edu>.
Hi Everyone,
Apologies for my late inactivity on the mailing list. I’ll try to cover what I’ve implemented so far, and the issues I’m facing and working on right now.
First of all, I now have an implementation of docker containers for filemgr, workflow, and resmgr. I verified that all these are starting up with no exceptions, and fixed environment variable issues.
Next, I started creating a docker image for the solr catalog. I initially used solr 8 (latest version) and then found various issues and downgraded it until I was able to start the solr server, which is solr version 5.5.5.
I also created a docker image for ops ui, but couldn’t get all interfaces working as expected, and have a lot of exceptions being thrown when visiting certain pages. I’m working on getting them resolved, and will post an update about these after tomorrow’s call with Imesha and Tom.
From last week, my goal was to create an end-to-end working implementation where I could ingest a file using filemgr_client, and see it appear on opsui. I made it up to the point where all components are running. I also created a docker compose file to facilitate exactly that.
I’ll keep working on getting the end-to-end-working implementation, and once I do I will create a pull request on the OODT repository. Please bear with me if it’s taking more time than expected. The exception logs show that certain log files and XML are missing (which is causing the exceptions), and I’m working on fixing them right now.
I welcome any comments or suggestions on how to tackle these issues better. :)
==================
### THIS EXCEPTION IS THROWN WHEN I GOTO “FILE CATALOG BROWSE” IN OPSUI
opsui_1 | ERROR - Objects - Error serializing object class org.apache.oodt.pcs.opsui.TypesPage [object=[Page class = org.apache.oodt.pcs.opsui.TypesPage, id = 4, version = 0]]
opsui_1 | org.apache.wicket.util.io.SerializableChecker$WicketNotSerializableException: Unable to serialize class: org.apache.oodt.cas.filemgr.structs.ProductType
opsui_1 | Field hierarchy is:
opsui_1 | 4 [class=org.apache.oodt.pcs.opsui.TypesPage, path=4]
opsui_1 | private java.lang.Object org.apache.wicket.MarkupContainer.children [class=[Ljava.lang.Object;]
opsui_1 | private org.apache.wicket.markup.html.link.PopupSettings org.apache.wicket.markup.html.link.ExternalLink.popupSettings[8] [class=org.apache.oodt.cas.webcomponents.filemgr.browser.types.Types, path=4:types_component]
opsui_1 | private java.lang.Object org.apache.wicket.MarkupContainer.children [class=org.apache.oodt.cas.webcomponents.filemgr.browser.types.Types$2, path=4:types_component:cas_fm_browser_ptype_table_rows]
opsui_1 | java.lang.Object org.apache.wicket.Component.data [class=org.apache.wicket.model.util.WildcardListModel]
opsui_1 | private java.lang.Object org.apache.wicket.model.util.GenericBaseModel.object [class=java.util.Vector]
opsui_1 | private java.lang.Object org.apache.wicket.model.util.GenericBaseModel.object[write:1] [class=[Ljava.lang.Object;]
opsui_1 | private java.lang.Object org.apache.wicket.model.util.GenericBaseModel.object[write:1][0] [class=org.apache.oodt.cas.webcomponents.filemgr.browser.types.Types$TypeCountTuple]
opsui_1 | private java.util.List org.apache.oodt.cas.webcomponents.filemgr.browser.types.Types$TypeCountTuple.types [class=java.util.Vector]
opsui_1 | private java.util.List org.apache.oodt.cas.webcomponents.filemgr.browser.types.Types$TypeCountTuple.types[write:1] [class=[Ljava.lang.Object;]
opsui_1 | private java.util.List org.apache.oodt.cas.webcomponents.filemgr.browser.types.Types$TypeCountTuple.types[write:1][0] [class=org.apache.oodt.cas.filemgr.structs.ProductType] <----- field that is not serializable
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:386)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:432)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.access$500(SerializableChecker.java:64)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker$1InterceptingObjectOutputStream.replaceObject(SerializableChecker.java:535)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1143)
opsui_1 | at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
opsui_1 | at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1706)
opsui_1 | at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:481)
opsui_1 | at java.util.Vector.writeObject(Vector.java:1100)
opsui_1 | at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
opsui_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
opsui_1 | at java.lang.reflect.Method.invoke(Method.java:606)
opsui_1 | at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1131)
opsui_1 | at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
opsui_1 | at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
opsui_1 | at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:542)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.checkFields(SerializableChecker.java:656)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:579)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:432)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.access$500(SerializableChecker.java:64)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker$1InterceptingObjectOutputStream.replaceObject(SerializableChecker.java:535)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1143)
opsui_1 | at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
opsui_1 | at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1706)
opsui_1 | at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:481)
opsui_1 | at java.util.Vector.writeObject(Vector.java:1100)
opsui_1 | at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
opsui_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
opsui_1 | at java.lang.reflect.Method.invoke(Method.java:606)
opsui_1 | at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1131)
opsui_1 | at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
opsui_1 | at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
opsui_1 | at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:542)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.checkFields(SerializableChecker.java:656)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:579)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.checkFields(SerializableChecker.java:656)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:579)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.checkFields(SerializableChecker.java:656)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:579)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:432)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.checkFields(SerializableChecker.java:656)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:579)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
opsui_1 | at org.apache.wicket.util.io.SerializableChecker.writeObjectOverride(SerializableChecker.java:725)
opsui_1 | at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:343)
opsui_1 | at org.apache.wicket.util.io.IObjectStreamFactory$DefaultObjectStreamFactory$2.writeObjectOverride(IObjectStreamFactory.java:129)
opsui_1 | at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:343)
opsui_1 | at org.apache.wicket.util.lang.Objects.objectToByteArray(Objects.java:1124)
opsui_1 | at org.apache.wicket.protocol.http.pagestore.AbstractPageStore.serializePage(AbstractPageStore.java:203)
opsui_1 | at org.apache.wicket.protocol.http.pagestore.DiskPageStore.storePage(DiskPageStore.java:840)
opsui_1 | at org.apache.wicket.protocol.http.SecondLevelCacheSessionStore$SecondLevelCachePageMap.put(SecondLevelCacheSessionStore.java:332)
opsui_1 | at org.apache.wicket.Session.requestDetached(Session.java:1435)
opsui_1 | at org.apache.wicket.RequestCycle.detach(RequestCycle.java:1176)
opsui_1 | at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1462)
opsui_1 | at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
opsui_1 | at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:486)
opsui_1 | at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:319)
opsui_1 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
opsui_1 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
opsui_1 | at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
opsui_1 | at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
opsui_1 | at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
opsui_1 | at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
opsui_1 | at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
opsui_1 | at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
opsui_1 | at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
opsui_1 | at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
opsui_1 | at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1152)
opsui_1 | at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
opsui_1 | at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2527)
opsui_1 | at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2516)
opsui_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
opsui_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
opsui_1 | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
opsui_1 | at java.lang.Thread.run(Thread.java:748)
opsui_1 | Caused by: java.io.NotSerializableException: org.apache.oodt.cas.filemgr.structs.ProductType
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
opsui_1 | at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
opsui_1 | at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
opsui_1 | at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1706)
opsui_1 | at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:481)
opsui_1 | at java.util.Vector.writeObject(Vector.java:1100)
opsui_1 | at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
opsui_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
opsui_1 | at java.lang.reflect.Method.invoke(Method.java:606)
opsui_1 | at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1131)
opsui_1 | at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
opsui_1 | at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
opsui_1 | at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
opsui_1 | at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
opsui_1 | at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
opsui_1 | at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
opsui_1 | at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
opsui_1 | at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1706)
opsui_1 | at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:481)
opsui_1 | at java.util.Vector.writeObject(Vector.java:1100)
opsui_1 | at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
opsui_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
opsui_1 | at java.lang.reflect.Method.invoke(Method.java:606)
opsui_1 | at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1131)
opsui_1 | at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
opsui_1 | at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
opsui_1 | at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
opsui_1 | at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
opsui_1 | at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
opsui_1 | at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
opsui_1 | at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:440)
opsui_1 | at org.apache.wicket.Component.writeObject(Component.java:4700)
opsui_1 | at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
opsui_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
opsui_1 | at java.lang.reflect.Method.invoke(Method.java:606)
opsui_1 | at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1131)
opsui_1 | at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
opsui_1 | at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
opsui_1 | at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
opsui_1 | at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
opsui_1 | at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
opsui_1 | at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
opsui_1 | at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
opsui_1 | at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
opsui_1 | at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
opsui_1 | at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
opsui_1 | at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
opsui_1 | at org.apache.wicket.util.io.IObjectStreamFactory$DefaultObjectStreamFactory$2.writeObjectOverride(IObjectStreamFactory.java:121)
opsui_1 | ... 29 more
====================
Best Regards,
Yasith Jayawardana
------------------------------------------------
Yasith Jayawardana
PhD Student | Old Dominion University
phone: +1 757 698 8978
On Jun 6, 2019, at 10:48 AM, Imesha Sudasingha <im...@apache.org>> wrote:
Hi Chris,
Yes. That explains it.
Hi Yasith,
Therefore, let's not make changes to the structure of those scripts. But,
if you see anything missing (ex: logging properties file is not passed as a
vm argument) should be addressed. If you found any such mismatch, let's see
if that can be synced.
Imesha
On Thu, 6 Jun 2019 at 12:01, Chris Mattmann <ma...@apache.org>> wrote:
The requirements for the scripts are:
Filemanager as a component must work without docker, and without
RADIX, so any changes to that core script need to satisfy that.
Additionally Filemanager has no dependencies on workflow, other
envs, or resource, etc., so any updates to the FM script in the core
FM component would need to respect that
RADIX on the other hand, depends on everything, env variables, everything
else, the core package. Given that, it makes sense to have its filemgr
script
be different and part of something bigger.
Does this make sense?
Cheers,
Chris
From: Imesha Sudasingha <im...@apache.org>>
Reply-To: <de...@oodt.apache.org>>
Date: Wednesday, June 5, 2019 at 11:08 PM
To: dev <de...@oodt.apache.org>>, Chris Mattmann <ma...@apache.org>>
Subject: Re: [GSOC] OODT docker build meeting summary
Hi Yasith,
Thanks for the meeting notes. Regarding those two questions,
1. Yes, I have noticed that those scripts are out of sync. But, we cannot
make those two even directly since, the script within Radix is supposed to
be run by another shell script which exports some variables like OODT_HOME,
RUN_JAVA. In contrast, shell script within filmgr module is supposed to be
run in standalone mode. Therefore, if we are to sync those two scripts,
only possibility is to add any missing lines and etc. @Chris Mattmann
<ma...@apache.org>> what is your opinion on this?
2. They should definitely be updated. Could you create Jira issues for
those pom files (one issue containing all pom files will be enough)? It
will be great if you can send a fix for that as well.
Thanks,
Imesha
On Tue, 4 Jun 2019 at 22:20, Jayawardana, Yasith <ya...@cs.odu.edu>>
wrote:
Hi All,
Here are the meeting minutes from the discussion today with Imesha, and
the discussion last week with Tom.
05/28 - GSOC Meeting [with Tom] (9 AM EST)
* Status
* I have created the DMB repository before the meeting as first
step, and plan to add the dockerfile-maven plugin next.
* Discussion
* Set default docker image name to project build name and the
default tag to project version
* Run the shell script for file manager in docker container and see
fix errors.
* Also create a diagram indicating which components depend on the
others so that others can benefit.
* Goal for the week is to get to a stage where dockerfile maven
generates the docker image, and filemgr can run in it without errors
06/04 - GSOC Meeting [with Imesha] (9 AM EST)
* Status
* I created a dockerfile for filemgr and made a few changes to the
filemgr script, so that filemgr start works inside docker as well. Now the
maven build process generates a docker image from the dockerfile at the
package phase, and I’m able to start it without encountering any issues.
* Discussion
* Instead of running the shell script, run the command directly in
Dockerfile. Makes things easier.
* For the development purposes, create the docker images on the
default build profile itself. Later move them to a separate profile.
* Once the image works and confirmed running well, discuss with
others on which directories should be mounted in volumes.
There are also a few concerns that I forgot to discuss on today’s meeting
which I want to mention.
* I saw that filemgr shell script inside radix is different from the
one in core distribution. Is this intentional or is it a bug?
* I saw some plugin versions in pom files are beta (especially
maven-assembly-plugin). Shouldn’t these be updated to non-beta versions?
Appreciate any suggestions on these.
Best Regards,
Yasith Jayawardana
On May 27, 2019, at 10:04 PM, Jayawardana, Yasith <ya...@cs.odu.edu>
<ma...@cs.odu.edu>> wrote:
[MEETING NOTES - 05/21/2019]
* As the first step of the project, create a new repository from the
RADIX build.
* Customize it such that it creates Docker Images at “mvn install”, and
publishes them to DockerHub at “mvn deploy”.
* To achieve this, the first component that should be containerized is
OODT file manager, as it contains the least dependencies on other
components of OODT.
* Maven assembly plugin is used at the "mvn build" stage. Therefore,
the docker image creation should happen in a later stage. i.e. “mvn
install” stage.
* Use the tar.gz file created from the maven assembly plugin as the
source for generating the docker image for that component.
* The docker image creation should not be enabled by default, and
should be designed as an optional output. User should be able to configure
the credentials of DockerHub, etc somewhere in the build step.
Apologies for taking long to post this update.
[PROGRESS UPDATE]
So far, I created a new repository (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fyasithmilinda%2Foodt-dmb&data=02%7C01%7Cyasith%40cs.odu.edu%7C7a9baa77e8274eaa1bc308d6ea8e233f%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636954293538526705&sdata=WC2f2EN32e%2FKUSgbf9%2BYlL0OuUZq66D8eqnH6CsrAss%3D&reserved=0
)
to implement the project and add the additional build steps needed to
create/publish docker images. As the implementation becomes stable, the
changed/added files will be added/modified in OODT source code, and this
repo could be used as an example implementation of it.
If there’s anything that I've misunderstood and/or needs to be changed, do
let me know. Thanks!
Best,
Yasith
On May 14, 2019, at 11:17 AM, Tom Barber <to...@spicule.co.uk><mailto:
tom@spicule.co.uk<ma...@spicule.co.uk>> wrote:
Yasith, to get you off to a good start in terms of open source development.
Can you please file the issues you expect to work on in the short term in
Jira so we can track them. If you do it it’ll ensure that you have enough
knowledge rather than us doing it and them not making sense.
Tom
On 14 May 2019 at 15:12:43, Tom Barber (tom@spicule.co.uk<ma...@spicule.co.uk><mailto:
tom@spicule.co.uk<ma...@spicule.co.uk>) wrote:
Just a quick recap from our call with Yasith earlier today:
Had a good chat with Yasith about how RADIX works and how people build out
OODT applications from RADIX (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOODT%2FRADiX%2BPowered%2BBy%2BOODT&data=02%7C01%7Cyasith%40cs.odu.edu%7C7a9baa77e8274eaa1bc308d6ea8e233f%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636954293538526705&sdata=7PwkSlnqWAU1RbPXZX%2BmTF530KyOOiNdkz5G%2FRJXDQI%3D&reserved=0
)
General Docker build plan is as follows:
* Users can deploy a standard RADIX build and optionally choose the Docker
profile at build time to build docker images from it
* Docker will be optional not standard
* RADIX will allow users to define the image tag names for easy tagging and
pushing to docker hub or similar
* In the GSOC version we will focus on single deployments of each component
not clustered up using ZK or similar we will stick that in the TODO list
Once built the end target is Kubernetes, both MiniKube and full Kubernetes.
Docker Compose may be used for interim testing and validation.
A few deploy methods exist for validation:
Standard deploy yams for K8S
Helm Chart
Or via a custom Kubernetes Operator similar to ArangoDB (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb&data=02%7C01%7Cyasith%40cs.odu.edu%7C7a9baa77e8274eaa1bc308d6ea8e233f%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636954293538526705&sdata=%2BoyNdtvvOaSQ39PtfylQKKLdMhRIaYFIe%2FYfC2RG%2BfI%3D&reserved=0
)
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: [GSOC] OODT docker build meeting summary
Posted by Imesha Sudasingha <im...@apache.org>.
Hi Chris,
Yes. That explains it.
Hi Yasith,
Therefore, let's not make changes to the structure of those scripts. But,
if you see anything missing (ex: logging properties file is not passed as a
vm argument) should be addressed. If you found any such mismatch, let's see
if that can be synced.
Imesha
On Thu, 6 Jun 2019 at 12:01, Chris Mattmann <ma...@apache.org> wrote:
> The requirements for the scripts are:
>
>
> Filemanager as a component must work without docker, and without
> RADIX, so any changes to that core script need to satisfy that.
> Additionally Filemanager has no dependencies on workflow, other
> envs, or resource, etc., so any updates to the FM script in the core
> FM component would need to respect that
> RADIX on the other hand, depends on everything, env variables, everything
> else, the core package. Given that, it makes sense to have its filemgr
> script
> be different and part of something bigger.
>
>
> Does this make sense?
>
>
>
> Cheers,
>
> Chris
>
>
>
>
>
>
>
>
>
> From: Imesha Sudasingha <im...@apache.org>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, June 5, 2019 at 11:08 PM
> To: dev <de...@oodt.apache.org>, Chris Mattmann <ma...@apache.org>
> Subject: Re: [GSOC] OODT docker build meeting summary
>
>
>
> Hi Yasith,
>
>
>
> Thanks for the meeting notes. Regarding those two questions,
>
>
>
> 1. Yes, I have noticed that those scripts are out of sync. But, we cannot
>
> make those two even directly since, the script within Radix is supposed to
>
> be run by another shell script which exports some variables like OODT_HOME,
>
> RUN_JAVA. In contrast, shell script within filmgr module is supposed to be
>
> run in standalone mode. Therefore, if we are to sync those two scripts,
>
> only possibility is to add any missing lines and etc. @Chris Mattmann
>
> <ma...@apache.org> what is your opinion on this?
>
>
>
> 2. They should definitely be updated. Could you create Jira issues for
>
> those pom files (one issue containing all pom files will be enough)? It
>
> will be great if you can send a fix for that as well.
>
>
>
> Thanks,
>
> Imesha
>
>
>
> On Tue, 4 Jun 2019 at 22:20, Jayawardana, Yasith <ya...@cs.odu.edu>
> wrote:
>
>
>
> Hi All,
>
>
>
> Here are the meeting minutes from the discussion today with Imesha, and
>
> the discussion last week with Tom.
>
>
>
>
>
> 05/28 - GSOC Meeting [with Tom] (9 AM EST)
>
>
>
> * Status
>
> * I have created the DMB repository before the meeting as first
>
> step, and plan to add the dockerfile-maven plugin next.
>
> * Discussion
>
> * Set default docker image name to project build name and the
>
> default tag to project version
>
> * Run the shell script for file manager in docker container and see
>
> fix errors.
>
> * Also create a diagram indicating which components depend on the
>
> others so that others can benefit.
>
> * Goal for the week is to get to a stage where dockerfile maven
>
> generates the docker image, and filemgr can run in it without errors
>
>
>
>
>
> 06/04 - GSOC Meeting [with Imesha] (9 AM EST)
>
>
>
> * Status
>
> * I created a dockerfile for filemgr and made a few changes to the
>
> filemgr script, so that filemgr start works inside docker as well. Now the
>
> maven build process generates a docker image from the dockerfile at the
>
> package phase, and I’m able to start it without encountering any issues.
>
> * Discussion
>
> * Instead of running the shell script, run the command directly in
>
> Dockerfile. Makes things easier.
>
> * For the development purposes, create the docker images on the
>
> default build profile itself. Later move them to a separate profile.
>
> * Once the image works and confirmed running well, discuss with
>
> others on which directories should be mounted in volumes.
>
>
>
> There are also a few concerns that I forgot to discuss on today’s meeting
>
> which I want to mention.
>
>
>
> * I saw that filemgr shell script inside radix is different from the
>
> one in core distribution. Is this intentional or is it a bug?
>
> * I saw some plugin versions in pom files are beta (especially
>
> maven-assembly-plugin). Shouldn’t these be updated to non-beta versions?
>
>
>
> Appreciate any suggestions on these.
>
>
>
> Best Regards,
>
> Yasith Jayawardana
>
>
>
>
>
> On May 27, 2019, at 10:04 PM, Jayawardana, Yasith <yasith@cs.odu.edu
>
> <ma...@cs.odu.edu>> wrote:
>
>
>
> [MEETING NOTES - 05/21/2019]
>
>
>
>
>
> * As the first step of the project, create a new repository from the
>
> RADIX build.
>
> * Customize it such that it creates Docker Images at “mvn install”, and
>
> publishes them to DockerHub at “mvn deploy”.
>
> * To achieve this, the first component that should be containerized is
>
> OODT file manager, as it contains the least dependencies on other
>
> components of OODT.
>
> * Maven assembly plugin is used at the "mvn build" stage. Therefore,
>
> the docker image creation should happen in a later stage. i.e. “mvn
>
> install” stage.
>
> * Use the tar.gz file created from the maven assembly plugin as the
>
> source for generating the docker image for that component.
>
> * The docker image creation should not be enabled by default, and
>
> should be designed as an optional output. User should be able to configure
>
> the credentials of DockerHub, etc somewhere in the build step.
>
>
>
> Apologies for taking long to post this update.
>
>
>
> [PROGRESS UPDATE]
>
> So far, I created a new repository (
>
>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fyasithmilinda%2Foodt-dmb&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=8DNciBssR3lnur1CEMiXuTJjqQxl1NZUd7XL6iN3cPg%3D&reserved=0
> )
>
> to implement the project and add the additional build steps needed to
>
> create/publish docker images. As the implementation becomes stable, the
>
> changed/added files will be added/modified in OODT source code, and this
>
> repo could be used as an example implementation of it.
>
>
>
>
>
> If there’s anything that I've misunderstood and/or needs to be changed, do
>
> let me know. Thanks!
>
>
>
> Best,
>
> Yasith
>
>
>
> On May 14, 2019, at 11:17 AM, Tom Barber <tom@spicule.co.uk<mailto:
>
> tom@spicule.co.uk><ma...@spicule.co.uk>> wrote:
>
>
>
> Yasith, to get you off to a good start in terms of open source development.
>
> Can you please file the issues you expect to work on in the short term in
>
> Jira so we can track them. If you do it it’ll ensure that you have enough
>
> knowledge rather than us doing it and them not making sense.
>
>
>
> Tom
>
>
>
>
>
> On 14 May 2019 at 15:12:43, Tom Barber (tom@spicule.co.uk<mailto:
>
> tom@spicule.co.uk><ma...@spicule.co.uk>) wrote:
>
>
>
> Just a quick recap from our call with Yasith earlier today:
>
>
>
> Had a good chat with Yasith about how RADIX works and how people build out
>
> OODT applications from RADIX (
>
>
>
>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOODT%2FRADiX%2BPowered%2BBy%2BOODT&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=9hbMVRs1dsT2PB08QFS69IYc88cgKWjzSTITUn%2B9Njg%3D&reserved=0
>
> )
>
>
>
> General Docker build plan is as follows:
>
>
>
> * Users can deploy a standard RADIX build and optionally choose the Docker
>
> profile at build time to build docker images from it
>
> * Docker will be optional not standard
>
> * RADIX will allow users to define the image tag names for easy tagging and
>
> pushing to docker hub or similar
>
> * In the GSOC version we will focus on single deployments of each component
>
> not clustered up using ZK or similar we will stick that in the TODO list
>
>
>
> Once built the end target is Kubernetes, both MiniKube and full Kubernetes.
>
> Docker Compose may be used for interim testing and validation.
>
>
>
> A few deploy methods exist for validation:
>
>
>
> Standard deploy yams for K8S
>
> Helm Chart
>
> Or via a custom Kubernetes Operator similar to ArangoDB (
>
>
>
>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=mt2pG%2BmMTcAd2FJuXtn%2FHwro090o4TBb6Um%2Fi1rzAvA%3D&reserved=0
>
> )
>
>
>
> --
>
>
>
>
>
> Spicule Limited is registered in England & Wales. Company Number:
>
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
>
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
>
>
>
>
>
> All engagements
>
> are subject to Spicule Terms and Conditions of Business. This email and its
>
> contents are intended solely for the individual to whom it is addressed and
>
> may contain information that is confidential, privileged or otherwise
>
> protected from disclosure, distributing or copying. Any views or opinions
>
> presented in this email are solely those of the author and do not
>
> necessarily represent those of Spicule Limited. The company accepts no
>
> liability for any damage caused by any virus transmitted by this email. If
>
> you have received this message in error, please notify us immediately by
>
> reply email before deleting it from your system. Service of legal notice
>
> cannot be effected on Spicule Limited by email.
>
>
>
>
>
>
>
>
>
>
Re: [GSOC] OODT docker build meeting summary
Posted by Chris Mattmann <ma...@apache.org>.
The requirements for the scripts are:
Filemanager as a component must work without docker, and without
RADIX, so any changes to that core script need to satisfy that.
Additionally Filemanager has no dependencies on workflow, other
envs, or resource, etc., so any updates to the FM script in the core
FM component would need to respect that
RADIX on the other hand, depends on everything, env variables, everything
else, the core package. Given that, it makes sense to have its filemgr script
be different and part of something bigger.
Does this make sense?
Cheers,
Chris
From: Imesha Sudasingha <im...@apache.org>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, June 5, 2019 at 11:08 PM
To: dev <de...@oodt.apache.org>, Chris Mattmann <ma...@apache.org>
Subject: Re: [GSOC] OODT docker build meeting summary
Hi Yasith,
Thanks for the meeting notes. Regarding those two questions,
1. Yes, I have noticed that those scripts are out of sync. But, we cannot
make those two even directly since, the script within Radix is supposed to
be run by another shell script which exports some variables like OODT_HOME,
RUN_JAVA. In contrast, shell script within filmgr module is supposed to be
run in standalone mode. Therefore, if we are to sync those two scripts,
only possibility is to add any missing lines and etc. @Chris Mattmann
<ma...@apache.org> what is your opinion on this?
2. They should definitely be updated. Could you create Jira issues for
those pom files (one issue containing all pom files will be enough)? It
will be great if you can send a fix for that as well.
Thanks,
Imesha
On Tue, 4 Jun 2019 at 22:20, Jayawardana, Yasith <ya...@cs.odu.edu> wrote:
Hi All,
Here are the meeting minutes from the discussion today with Imesha, and
the discussion last week with Tom.
05/28 - GSOC Meeting [with Tom] (9 AM EST)
* Status
* I have created the DMB repository before the meeting as first
step, and plan to add the dockerfile-maven plugin next.
* Discussion
* Set default docker image name to project build name and the
default tag to project version
* Run the shell script for file manager in docker container and see
fix errors.
* Also create a diagram indicating which components depend on the
others so that others can benefit.
* Goal for the week is to get to a stage where dockerfile maven
generates the docker image, and filemgr can run in it without errors
06/04 - GSOC Meeting [with Imesha] (9 AM EST)
* Status
* I created a dockerfile for filemgr and made a few changes to the
filemgr script, so that filemgr start works inside docker as well. Now the
maven build process generates a docker image from the dockerfile at the
package phase, and I’m able to start it without encountering any issues.
* Discussion
* Instead of running the shell script, run the command directly in
Dockerfile. Makes things easier.
* For the development purposes, create the docker images on the
default build profile itself. Later move them to a separate profile.
* Once the image works and confirmed running well, discuss with
others on which directories should be mounted in volumes.
There are also a few concerns that I forgot to discuss on today’s meeting
which I want to mention.
* I saw that filemgr shell script inside radix is different from the
one in core distribution. Is this intentional or is it a bug?
* I saw some plugin versions in pom files are beta (especially
maven-assembly-plugin). Shouldn’t these be updated to non-beta versions?
Appreciate any suggestions on these.
Best Regards,
Yasith Jayawardana
On May 27, 2019, at 10:04 PM, Jayawardana, Yasith <yasith@cs.odu.edu
<ma...@cs.odu.edu>> wrote:
[MEETING NOTES - 05/21/2019]
* As the first step of the project, create a new repository from the
RADIX build.
* Customize it such that it creates Docker Images at “mvn install”, and
publishes them to DockerHub at “mvn deploy”.
* To achieve this, the first component that should be containerized is
OODT file manager, as it contains the least dependencies on other
components of OODT.
* Maven assembly plugin is used at the "mvn build" stage. Therefore,
the docker image creation should happen in a later stage. i.e. “mvn
install” stage.
* Use the tar.gz file created from the maven assembly plugin as the
source for generating the docker image for that component.
* The docker image creation should not be enabled by default, and
should be designed as an optional output. User should be able to configure
the credentials of DockerHub, etc somewhere in the build step.
Apologies for taking long to post this update.
[PROGRESS UPDATE]
So far, I created a new repository (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fyasithmilinda%2Foodt-dmb&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=8DNciBssR3lnur1CEMiXuTJjqQxl1NZUd7XL6iN3cPg%3D&reserved=0)
to implement the project and add the additional build steps needed to
create/publish docker images. As the implementation becomes stable, the
changed/added files will be added/modified in OODT source code, and this
repo could be used as an example implementation of it.
If there’s anything that I've misunderstood and/or needs to be changed, do
let me know. Thanks!
Best,
Yasith
On May 14, 2019, at 11:17 AM, Tom Barber <tom@spicule.co.uk<mailto:
tom@spicule.co.uk><ma...@spicule.co.uk>> wrote:
Yasith, to get you off to a good start in terms of open source development.
Can you please file the issues you expect to work on in the short term in
Jira so we can track them. If you do it it’ll ensure that you have enough
knowledge rather than us doing it and them not making sense.
Tom
On 14 May 2019 at 15:12:43, Tom Barber (tom@spicule.co.uk<mailto:
tom@spicule.co.uk><ma...@spicule.co.uk>) wrote:
Just a quick recap from our call with Yasith earlier today:
Had a good chat with Yasith about how RADIX works and how people build out
OODT applications from RADIX (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOODT%2FRADiX%2BPowered%2BBy%2BOODT&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=9hbMVRs1dsT2PB08QFS69IYc88cgKWjzSTITUn%2B9Njg%3D&reserved=0
)
General Docker build plan is as follows:
* Users can deploy a standard RADIX build and optionally choose the Docker
profile at build time to build docker images from it
* Docker will be optional not standard
* RADIX will allow users to define the image tag names for easy tagging and
pushing to docker hub or similar
* In the GSOC version we will focus on single deployments of each component
not clustered up using ZK or similar we will stick that in the TODO list
Once built the end target is Kubernetes, both MiniKube and full Kubernetes.
Docker Compose may be used for interim testing and validation.
A few deploy methods exist for validation:
Standard deploy yams for K8S
Helm Chart
Or via a custom Kubernetes Operator similar to ArangoDB (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=mt2pG%2BmMTcAd2FJuXtn%2FHwro090o4TBb6Um%2Fi1rzAvA%3D&reserved=0
)
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: [GSOC] OODT docker build meeting summary
Posted by Imesha Sudasingha <im...@apache.org>.
Hi Yasith,
Thanks for the meeting notes. Regarding those two questions,
1. Yes, I have noticed that those scripts are out of sync. But, we cannot
make those two even directly since, the script within Radix is supposed to
be run by another shell script which exports some variables like OODT_HOME,
RUN_JAVA. In contrast, shell script within filmgr module is supposed to be
run in standalone mode. Therefore, if we are to sync those two scripts,
only possibility is to add any missing lines and etc. @Chris Mattmann
<ma...@apache.org> what is your opinion on this?
2. They should definitely be updated. Could you create Jira issues for
those pom files (one issue containing all pom files will be enough)? It
will be great if you can send a fix for that as well.
Thanks,
Imesha
On Tue, 4 Jun 2019 at 22:20, Jayawardana, Yasith <ya...@cs.odu.edu> wrote:
> Hi All,
>
> Here are the meeting minutes from the discussion today with Imesha, and
> the discussion last week with Tom.
>
>
> 05/28 - GSOC Meeting [with Tom] (9 AM EST)
>
> * Status
> * I have created the DMB repository before the meeting as first
> step, and plan to add the dockerfile-maven plugin next.
> * Discussion
> * Set default docker image name to project build name and the
> default tag to project version
> * Run the shell script for file manager in docker container and see
> fix errors.
> * Also create a diagram indicating which components depend on the
> others so that others can benefit.
> * Goal for the week is to get to a stage where dockerfile maven
> generates the docker image, and filemgr can run in it without errors
>
>
> 06/04 - GSOC Meeting [with Imesha] (9 AM EST)
>
> * Status
> * I created a dockerfile for filemgr and made a few changes to the
> filemgr script, so that filemgr start works inside docker as well. Now the
> maven build process generates a docker image from the dockerfile at the
> package phase, and I’m able to start it without encountering any issues.
> * Discussion
> * Instead of running the shell script, run the command directly in
> Dockerfile. Makes things easier.
> * For the development purposes, create the docker images on the
> default build profile itself. Later move them to a separate profile.
> * Once the image works and confirmed running well, discuss with
> others on which directories should be mounted in volumes.
>
> There are also a few concerns that I forgot to discuss on today’s meeting
> which I want to mention.
>
> * I saw that filemgr shell script inside radix is different from the
> one in core distribution. Is this intentional or is it a bug?
> * I saw some plugin versions in pom files are beta (especially
> maven-assembly-plugin). Shouldn’t these be updated to non-beta versions?
>
> Appreciate any suggestions on these.
>
> Best Regards,
> Yasith Jayawardana
>
>
> On May 27, 2019, at 10:04 PM, Jayawardana, Yasith <yasith@cs.odu.edu
> <ma...@cs.odu.edu>> wrote:
>
> [MEETING NOTES - 05/21/2019]
>
>
> * As the first step of the project, create a new repository from the
> RADIX build.
> * Customize it such that it creates Docker Images at “mvn install”, and
> publishes them to DockerHub at “mvn deploy”.
> * To achieve this, the first component that should be containerized is
> OODT file manager, as it contains the least dependencies on other
> components of OODT.
> * Maven assembly plugin is used at the "mvn build" stage. Therefore,
> the docker image creation should happen in a later stage. i.e. “mvn
> install” stage.
> * Use the tar.gz file created from the maven assembly plugin as the
> source for generating the docker image for that component.
> * The docker image creation should not be enabled by default, and
> should be designed as an optional output. User should be able to configure
> the credentials of DockerHub, etc somewhere in the build step.
>
> Apologies for taking long to post this update.
>
> [PROGRESS UPDATE]
> So far, I created a new repository (
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fyasithmilinda%2Foodt-dmb&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=8DNciBssR3lnur1CEMiXuTJjqQxl1NZUd7XL6iN3cPg%3D&reserved=0)
> to implement the project and add the additional build steps needed to
> create/publish docker images. As the implementation becomes stable, the
> changed/added files will be added/modified in OODT source code, and this
> repo could be used as an example implementation of it.
>
>
> If there’s anything that I've misunderstood and/or needs to be changed, do
> let me know. Thanks!
>
> Best,
> Yasith
>
> On May 14, 2019, at 11:17 AM, Tom Barber <tom@spicule.co.uk<mailto:
> tom@spicule.co.uk><ma...@spicule.co.uk>> wrote:
>
> Yasith, to get you off to a good start in terms of open source development.
> Can you please file the issues you expect to work on in the short term in
> Jira so we can track them. If you do it it’ll ensure that you have enough
> knowledge rather than us doing it and them not making sense.
>
> Tom
>
>
> On 14 May 2019 at 15:12:43, Tom Barber (tom@spicule.co.uk<mailto:
> tom@spicule.co.uk><ma...@spicule.co.uk>) wrote:
>
> Just a quick recap from our call with Yasith earlier today:
>
> Had a good chat with Yasith about how RADIX works and how people build out
> OODT applications from RADIX (
>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOODT%2FRADiX%2BPowered%2BBy%2BOODT&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=9hbMVRs1dsT2PB08QFS69IYc88cgKWjzSTITUn%2B9Njg%3D&reserved=0
> )
>
> General Docker build plan is as follows:
>
> * Users can deploy a standard RADIX build and optionally choose the Docker
> profile at build time to build docker images from it
> * Docker will be optional not standard
> * RADIX will allow users to define the image tag names for easy tagging and
> pushing to docker hub or similar
> * In the GSOC version we will focus on single deployments of each component
> not clustered up using ZK or similar we will stick that in the TODO list
>
> Once built the end target is Kubernetes, both MiniKube and full Kubernetes.
> Docker Compose may be used for interim testing and validation.
>
> A few deploy methods exist for validation:
>
> Standard deploy yams for K8S
> Helm Chart
> Or via a custom Kubernetes Operator similar to ArangoDB (
>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=mt2pG%2BmMTcAd2FJuXtn%2FHwro090o4TBb6Um%2Fi1rzAvA%3D&reserved=0
> )
>
> --
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
> All engagements
> are subject to Spicule Terms and Conditions of Business. This email and its
> contents are intended solely for the individual to whom it is addressed and
> may contain information that is confidential, privileged or otherwise
> protected from disclosure, distributing or copying. Any views or opinions
> presented in this email are solely those of the author and do not
> necessarily represent those of Spicule Limited. The company accepts no
> liability for any damage caused by any virus transmitted by this email. If
> you have received this message in error, please notify us immediately by
> reply email before deleting it from your system. Service of legal notice
> cannot be effected on Spicule Limited by email.
>
>
>
Re: [GSOC] OODT docker build meeting summary
Posted by "Jayawardana, Yasith" <ya...@cs.odu.edu>.
Hi All,
Here are the meeting minutes from the discussion today with Imesha, and the discussion last week with Tom.
05/28 - GSOC Meeting [with Tom] (9 AM EST)
* Status
* I have created the DMB repository before the meeting as first step, and plan to add the dockerfile-maven plugin next.
* Discussion
* Set default docker image name to project build name and the default tag to project version
* Run the shell script for file manager in docker container and see fix errors.
* Also create a diagram indicating which components depend on the others so that others can benefit.
* Goal for the week is to get to a stage where dockerfile maven generates the docker image, and filemgr can run in it without errors
06/04 - GSOC Meeting [with Imesha] (9 AM EST)
* Status
* I created a dockerfile for filemgr and made a few changes to the filemgr script, so that filemgr start works inside docker as well. Now the maven build process generates a docker image from the dockerfile at the package phase, and I’m able to start it without encountering any issues.
* Discussion
* Instead of running the shell script, run the command directly in Dockerfile. Makes things easier.
* For the development purposes, create the docker images on the default build profile itself. Later move them to a separate profile.
* Once the image works and confirmed running well, discuss with others on which directories should be mounted in volumes.
There are also a few concerns that I forgot to discuss on today’s meeting which I want to mention.
* I saw that filemgr shell script inside radix is different from the one in core distribution. Is this intentional or is it a bug?
* I saw some plugin versions in pom files are beta (especially maven-assembly-plugin). Shouldn’t these be updated to non-beta versions?
Appreciate any suggestions on these.
Best Regards,
Yasith Jayawardana
On May 27, 2019, at 10:04 PM, Jayawardana, Yasith <ya...@cs.odu.edu>> wrote:
[MEETING NOTES - 05/21/2019]
* As the first step of the project, create a new repository from the RADIX build.
* Customize it such that it creates Docker Images at “mvn install”, and publishes them to DockerHub at “mvn deploy”.
* To achieve this, the first component that should be containerized is OODT file manager, as it contains the least dependencies on other components of OODT.
* Maven assembly plugin is used at the "mvn build" stage. Therefore, the docker image creation should happen in a later stage. i.e. “mvn install” stage.
* Use the tar.gz file created from the maven assembly plugin as the source for generating the docker image for that component.
* The docker image creation should not be enabled by default, and should be designed as an optional output. User should be able to configure the credentials of DockerHub, etc somewhere in the build step.
Apologies for taking long to post this update.
[PROGRESS UPDATE]
So far, I created a new repository (https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fyasithmilinda%2Foodt-dmb&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=8DNciBssR3lnur1CEMiXuTJjqQxl1NZUd7XL6iN3cPg%3D&reserved=0) to implement the project and add the additional build steps needed to create/publish docker images. As the implementation becomes stable, the changed/added files will be added/modified in OODT source code, and this repo could be used as an example implementation of it.
If there’s anything that I've misunderstood and/or needs to be changed, do let me know. Thanks!
Best,
Yasith
On May 14, 2019, at 11:17 AM, Tom Barber <to...@spicule.co.uk>> wrote:
Yasith, to get you off to a good start in terms of open source development.
Can you please file the issues you expect to work on in the short term in
Jira so we can track them. If you do it it’ll ensure that you have enough
knowledge rather than us doing it and them not making sense.
Tom
On 14 May 2019 at 15:12:43, Tom Barber (tom@spicule.co.uk<ma...@spicule.co.uk>) wrote:
Just a quick recap from our call with Yasith earlier today:
Had a good chat with Yasith about how RADIX works and how people build out
OODT applications from RADIX (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOODT%2FRADiX%2BPowered%2BBy%2BOODT&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=9hbMVRs1dsT2PB08QFS69IYc88cgKWjzSTITUn%2B9Njg%3D&reserved=0)
General Docker build plan is as follows:
* Users can deploy a standard RADIX build and optionally choose the Docker
profile at build time to build docker images from it
* Docker will be optional not standard
* RADIX will allow users to define the image tag names for easy tagging and
pushing to docker hub or similar
* In the GSOC version we will focus on single deployments of each component
not clustered up using ZK or similar we will stick that in the TODO list
Once built the end target is Kubernetes, both MiniKube and full Kubernetes.
Docker Compose may be used for interim testing and validation.
A few deploy methods exist for validation:
Standard deploy yams for K8S
Helm Chart
Or via a custom Kubernetes Operator similar to ArangoDB (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb&data=02%7C01%7Cyasith%40cs.odu.edu%7C562ed13a48344223267608d6e310e8ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636946059106151479&sdata=mt2pG%2BmMTcAd2FJuXtn%2FHwro090o4TBb6Um%2Fi1rzAvA%3D&reserved=0)
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: [GSOC] OODT docker build meeting summary
Posted by "Jayawardana, Yasith" <ya...@cs.odu.edu>.
[MEETING NOTES - 05/21/2019]
* As the first step of the project, create a new repository from the RADIX build.
* Customize it such that it creates Docker Images at “mvn install”, and publishes them to DockerHub at “mvn deploy”.
* To achieve this, the first component that should be containerized is OODT file manager, as it contains the least dependencies on other components of OODT.
* Maven assembly plugin is used at the "mvn build" stage. Therefore, the docker image creation should happen in a later stage. i.e. “mvn install” stage.
* Use the tar.gz file created from the maven assembly plugin as the source for generating the docker image for that component.
* The docker image creation should not be enabled by default, and should be designed as an optional output. User should be able to configure the credentials of DockerHub, etc somewhere in the build step.
Apologies for taking long to post this update.
[PROGRESS UPDATE]
So far, I created a new repository (https://github.com/yasithmilinda/oodt-dmb) to implement the project and add the additional build steps needed to create/publish docker images. As the implementation becomes stable, the changed/added files will be added/modified in OODT source code, and this repo could be used as an example implementation of it.
If there’s anything that I've misunderstood and/or needs to be changed, do let me know. Thanks!
Best,
Yasith
On May 14, 2019, at 11:17 AM, Tom Barber <to...@spicule.co.uk>> wrote:
Yasith, to get you off to a good start in terms of open source development.
Can you please file the issues you expect to work on in the short term in
Jira so we can track them. If you do it it’ll ensure that you have enough
knowledge rather than us doing it and them not making sense.
Tom
On 14 May 2019 at 15:12:43, Tom Barber (tom@spicule.co.uk<ma...@spicule.co.uk>) wrote:
Just a quick recap from our call with Yasith earlier today:
Had a good chat with Yasith about how RADIX works and how people build out
OODT applications from RADIX (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOODT%2FRADiX%2BPowered%2BBy%2BOODT&data=02%7C01%7Cyasith%40cs.odu.edu%7Cb68c0f2dc95a4254845f08d6d88039ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636934442576304882&sdata=IeQSq5o9ENrEu19VYbrLmjgdlmf2wPttWWb9pDqeoao%3D&reserved=0)
General Docker build plan is as follows:
* Users can deploy a standard RADIX build and optionally choose the Docker
profile at build time to build docker images from it
* Docker will be optional not standard
* RADIX will allow users to define the image tag names for easy tagging and
pushing to docker hub or similar
* In the GSOC version we will focus on single deployments of each component
not clustered up using ZK or similar we will stick that in the TODO list
Once built the end target is Kubernetes, both MiniKube and full Kubernetes.
Docker Compose may be used for interim testing and validation.
A few deploy methods exist for validation:
Standard deploy yams for K8S
Helm Chart
Or via a custom Kubernetes Operator similar to ArangoDB (
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb&data=02%7C01%7Cyasith%40cs.odu.edu%7Cb68c0f2dc95a4254845f08d6d88039ac%7C48bf86e811a24b8a8cb368d8be2227f3%7C0%7C0%7C636934442576304882&sdata=5N%2BhyHJMtVsshsTAGQUHZMNteipGgE%2FgKeM7NZvxHi8%3D&reserved=0)
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: [GSOC] OODT docker build meeting summary
Posted by Tom Barber <to...@spicule.co.uk>.
Yasith, to get you off to a good start in terms of open source development.
Can you please file the issues you expect to work on in the short term in
Jira so we can track them. If you do it it’ll ensure that you have enough
knowledge rather than us doing it and them not making sense.
Tom
On 14 May 2019 at 15:12:43, Tom Barber (tom@spicule.co.uk) wrote:
Just a quick recap from our call with Yasith earlier today:
Had a good chat with Yasith about how RADIX works and how people build out
OODT applications from RADIX (
https://cwiki.apache.org/confluence/display/OODT/RADiX+Powered+By+OODT)
General Docker build plan is as follows:
* Users can deploy a standard RADIX build and optionally choose the Docker
profile at build time to build docker images from it
* Docker will be optional not standard
* RADIX will allow users to define the image tag names for easy tagging and
pushing to docker hub or similar
* In the GSOC version we will focus on single deployments of each component
not clustered up using ZK or similar we will stick that in the TODO list
Once built the end target is Kubernetes, both MiniKube and full Kubernetes.
Docker Compose may be used for interim testing and validation.
A few deploy methods exist for validation:
Standard deploy yams for K8S
Helm Chart
Or via a custom Kubernetes Operator similar to ArangoDB (
https://github.com/arangodb/kube-arangodb)
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.