You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uima.apache.org by sc...@apache.org on 2020/03/18 17:27:34 UTC
svn commit: r1875373 - in /uima/site/trunk/uima-website:
docs/doc-uimaducc-whatitam.html docs/ducc-cloud.html
xdocs/doc-uimaducc-whatitam.xml xdocs/ducc-cloud.xml
Author: schor
Date: Wed Mar 18 17:27:33 2020
New Revision: 1875373
URL: http://svn.apache.org/viewvc?rev=1875373&view=rev
Log:
add info on DUCC and Cloud deployments
Added:
uima/site/trunk/uima-website/docs/ducc-cloud.html
uima/site/trunk/uima-website/xdocs/ducc-cloud.xml
Modified:
uima/site/trunk/uima-website/docs/doc-uimaducc-whatitam.html
uima/site/trunk/uima-website/xdocs/doc-uimaducc-whatitam.xml
Modified: uima/site/trunk/uima-website/docs/doc-uimaducc-whatitam.html
URL: http://svn.apache.org/viewvc/uima/site/trunk/uima-website/docs/doc-uimaducc-whatitam.html?rev=1875373&r1=1875372&r2=1875373&view=diff
==============================================================================
--- uima/site/trunk/uima-website/docs/doc-uimaducc-whatitam.html (original)
+++ uima/site/trunk/uima-website/docs/doc-uimaducc-whatitam.html Wed Mar 18 17:27:33 2020
@@ -252,7 +252,7 @@ This guide is intended to be a user orie
DUCC is a Linux cluster controller designed to scale out any UIMA pipeline
for high throughput collection processing jobs as well as for low latency
real-tme applications.
-Building on UIMA-AS, DUCC is particularly well suited to run large memory Java
+DUCC is particularly well suited to run large memory Java
analytics in multiple threads in order to fully utilize multicore machines.
DUCC manages the life cycle of all processes deployed across the cluster, including
non-UIMA processes such as tomcat servers or VNC sessions.
@@ -486,6 +486,26 @@ by adding a single parameter to the Job
</blockquote>
</td></tr>
</table>
+ <table class="subsectionTable">
+ <tr><td>
+
+
+
+ <a name="DUCC on Clouds">
+ <h2>DUCC on Clouds
+ </h2>
+ </a>
+ </td></tr>
+ <tr><td>
+ <blockquote class="subsectionBody">
+ <p>
+ DUCC has been running in various cloud deployments.
+ The documentation for this is still in-progress, but some information is
+ available <a target="_blank" rel="nofollow noopener" href="doc-ducc-cloud">here</a>.
+ </p>
+ </blockquote>
+ </td></tr>
+ </table>
<table class="subsectionTable">
<tr><td>
Added: uima/site/trunk/uima-website/docs/ducc-cloud.html
URL: http://svn.apache.org/viewvc/uima/site/trunk/uima-website/docs/ducc-cloud.html?rev=1875373&view=auto
==============================================================================
--- uima/site/trunk/uima-website/docs/ducc-cloud.html (added)
+++ uima/site/trunk/uima-website/docs/ducc-cloud.html Wed Mar 18 17:27:33 2020
@@ -0,0 +1,363 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "https://www.w3.org/TR/html4/loose.dtd">
+
+
+ <!-- ====================================================================== -->
+ <!-- GENERATED FILE, DO NOT EDIT, EDIT THE XML FILE IN xdocs INSTEAD! -->
+ <!-- ====================================================================== -->
+ <html>
+ <head>
+ <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
+ <style type="text/css">@import "stylesheets/base.css";</style>
+ <meta name="author" value="
+ Apache UIMA Team
+ ">
+ <meta name="email" value="dev@uima.apache.org">
+
+
+
+ <title>Apache UIMA - Running UIMA DUCC in a Cloud environment</title>
+
+ <!-- Begin Cookie Consent plugin by Silktide - https://silktide.com/cookieconsent -->
+ <!-- Commented out because implied consent is not compatible with GDPR -->
+ <!--
+ <script type="text/javascript">
+ window.cookieconsent_options = {"message":"This website uses cookies to ensure you get the best experience on our website","dismiss":"Got it!","learnMore":"More info","link":"https://uima.apache.org/privacy-policy.html","theme":"dark-bottom"};
+ </script>
+
+ <script type="text/javascript" src="/cookieconsent2/cookieconsent.min.js"></script>
+ -->
+ <!-- End Cookie Consent plugin -->
+
+ <!-- Begin Google Analytics -->
+ <!-- Commented out because GA requires consent according to GDPR -->
+ <!--
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-70846351-1', 'auto');
+ ga('set', 'anonymizeIp', true);
+ ga('send', 'pageview');
+
+ </script>
+ -->
+ <!-- End Google Analytics -->
+ </head>
+
+ <body>
+ <div class="topLogos">
+ <table border="0" width="100%" cellspacing="0">
+ <!-- TOP IMAGE -->
+ <tr>
+ <td align='LEFT'>
+ <a href="index.html">
+ <img style="border: 1px solid black;" src="./images/UIMA_banner2tlpTm.png" alt="UIMA project logo" border="0"/>
+ </a>
+ </td>
+ <td align='CENTER'>
+ <div class="pageBanner">Running UIMA DUCC in a Cloud environment</div>
+ </td>
+ <td align='RIGHT'>
+ <a href="https://www.apache.org">
+ <img src="./images/asf-logo-on-white-smallTm.png" alt="Apache UIMA" border="0"/>
+ </a>
+ </td>
+ </tr>
+ </table>
+ <hr noshade="" size="1"/>
+ </div>
+ <table border="0" width="100%" cellspacing="4">
+ <tr>
+ <td align='RIGHT' colspan="2">
+ <form method="get" action="https://www.google.com/search">
+ Search the site
+ <input type="text" name="q" size="25" maxlength="255" value="" />
+ <input type="hidden" name="sitesearch" value="https://uima.apache.org/" />
+ <input name="Search" value="Search Site" type="submit"/>
+ </form>
+ </td>
+ </tr>
+ <tr> <!-- LEFT SIDE NAVIGATION -->
+ <td width="20%" valign="top">
+
+
+
+
+
+
+ <!-- regular menu -->
+ <div class="navBar">
+ <br/>
+ <div class="navBarItem"> <div class="navPartHeading">General</div>
+ </div>
+ <div class="navBar">
+ <div class="navBarItem"> <a href="./index.html">Home</a>
+ </div>
+ <div class="navBarItem"> <a href="./downloads.cgi">Downloads</a>
+ </div>
+ <div class="navBarItem"> <a href="./documentation.html">Documentation</a>
+ </div>
+ <div class="navBarItem"> <a href="./news.html">News</a>
+ </div>
+ <div class="navBarItem"> <a href="./publications.html">Publications</a>
+ </div>
+ <br style="line-height: .5em"/>
+ <div class="navBarItem"> <a href="https://issues.apache.org/jira/browse/uima" target="_blank" rel="noopener">Issue tracker <img src="images/offsitelink.png"/></a>
+ </div>
+ <div class="navBarItem"> <a href="https://cwiki.apache.org/confluence/display/UIMA/" target="_blank" rel="noopener">Wiki <img src="images/offsitelink.png"/></a>
+ </div>
+ <br style="line-height: .5em"/>
+ <div class="navBarItem"> <a href="https://cwiki.apache.org/confluence/display/UIMA/Powered+by+Apache+UIMA" target="_blank" rel="noopener">Powered By UIMA <img src="images/offsitelink.png"/></a>
+ </div>
+ </div>
+ <br/>
+ <div class="navBarItem"> <div class="navPartHeading">Community</div>
+ </div>
+ <div class="navBar">
+ <div class="navBarItem"> <a href="./get-involved.html">Get Involved</a>
+ </div>
+ <div class="navBarItem"> <a href="./mail-lists.html">Mailing Lists</a>
+ </div>
+ <div class="navBarItem"> <a href="./contribution-policy.html">Contribution Policies</a>
+ </div>
+ <div class="navBarItem"> <a href="./faq.html">FAQ</a>
+ </div>
+ <div class="navBarItem"> <a href="./project-guidelines.html">Project Guidelines</a>
+ </div>
+ </div>
+ <br/>
+ <div class="navBarItem"> <div class="navPartHeading">Scaleout Frameworks</div>
+ </div>
+ <div class="navBar">
+ <div class="navBarItem"> <a href="./doc-uimaas-what.html">UIMA-AS</a>
+ </div>
+ <div class="navBarItem"> <a href="./doc-uimaducc-whatitam.html">UIMA-DUCC</a>
+ </div>
+ <div class="navBarItem"> <a href="./doc-uimaducc-demo.html">..Demo Page</a>
+ </div>
+ <div class="navBarItem"> <a href="http://uima-ducc-demo.apache.org:42133" target="_blank" rel="noopener">..Demo Live <img src="images/offsitelink.png"/></a>
+ </div>
+ </div>
+ <br/>
+ <div class="navBarItem"> <div class="navPartHeading">Components & Tools</div>
+ </div>
+ <div class="navBar">
+ <div class="navBarItem"> <a href="./sandbox.html#uima-addons-annotators">Annotators</a>
+ </div>
+ <div class="navBarItem"> <a href="./toolsServers.html">Tools & Servers</a>
+ </div>
+ <div class="navBarItem"> <a href="./sandbox.html">Addons and Sandbox</a>
+ </div>
+ <div class="navBarItem"> <a href="./ruta.html">UIMA Ruta</a>
+ </div>
+ <div class="navBarItem"> <a href="./uimafit.html">uimaFIT</a>
+ </div>
+ <div class="navBarItem"> <a href="./external-resources.html">External Resources</a>
+ </div>
+ </div>
+ <br/>
+ <div class="navBarItem"> <div class="navPartHeading">Development</div>
+ </div>
+ <div class="navBar">
+ <div class="navBarItem"> <a href="./dev-quick.html">Quick Start: building</a>
+ </div>
+ <div class="navBarItem"> <a href="./building-uima.html">Building from Source</a>
+ </div>
+ <div class="navBarItem"> <a href="./one-time-setup.html">One-time setups</a>
+ </div>
+ <div class="navBarItem"> <a href="./svn.html">Source Code</a>
+ </div>
+ <div class="navBarItem"> <a href="./distribution.html">Creating a Distribution</a>
+ </div>
+ <div class="navBarItem"> <a href="./release.html">Doing a UIMA release</a>
+ </div>
+ <div class="navBarItem"> <a href="https://www.apache.org/security/committers.html" target="_blank" rel="noopener">Doing a CVE (Apache) <img src="images/offsitelink.png"/></a>
+ </div>
+ <div class="navBarItem"> <a href="./eclipse-update-site.html">Eclipse Update Sites</a>
+ </div>
+ <div class="navBarItem"> <a href="./git.html">GIT</a>
+ </div>
+ <div class="navBarItem"> <a href="./codeConventions.html">Code Conventions</a>
+ </div>
+ <div class="navBarItem"> <a href="./uima-specification.html">UIMA Specification (OASIS)</a>
+ </div>
+ <div class="navBarItem"> <a href="./team-list.html">Project Team</a>
+ </div>
+ <div class="navBarItem"> <a href="./maven-design.html">Maven Use</a>
+ </div>
+ <div class="navBarItem"> <a href="./updating-website.html">Updating this Website</a>
+ </div>
+ </div>
+ <br/>
+ <div class="navBarItem"> <div class="navPartHeading">Events and Conferences</div>
+ </div>
+ <div class="navBar">
+ <div class="navBarItem"> <a href="./coling14.html">COLING 2014</a>
+ </div>
+ <div class="navBarItem"> <a href="./gscl13.html">GSCL 2013</a>
+ </div>
+ <div class="navBarItem"> <a href="./iks09.html">IKS 2009</a>
+ </div>
+ <div class="navBarItem"> <a href="./gscl09.html">GSCL 2009</a>
+ </div>
+ <div class="navBarItem"> <a href="./lsm09.html">LSM 2009</a>
+ </div>
+ <div class="navBarItem"> <a href="./lrec08.html">LREC 2008</a>
+ </div>
+ <div class="navBarItem"> <a href="./gldv07.html">GLDV 2007</a>
+ </div>
+ </div>
+ <br/>
+ <div class="navBarItem"> <div class="navPartHeading">ASF</div>
+ </div>
+ <div class="navBar">
+ <div class="navBarItem"> <a href="https://www.apache.org/licenses/" target="_blank" rel="noopener">License <img src="images/offsitelink.png"/></a>
+ </div>
+ <div class="navBarItem"> <a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener">ASF Sponsors <img src="images/offsitelink.png"/></a>
+ </div>
+ <div class="navBarItem"> <a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener">ASF Sponsorship <img src="images/offsitelink.png"/></a>
+ </div>
+ <div class="navBarItem"> <a href="./security_report">Security</a>
+ </div>
+ </div>
+ </div>
+ </td>
+ <td width="80%" align="left" valign="top">
+ <div class="sectionTable">
+ <table class="sectionTable">
+ <tr><td>
+ <a name="DUCC in Clouds"><h1><img src="images/UIMA_4sq50tightCropSolid.png"/> DUCC in Clouds</h1></a>
+ </td></tr>
+ <tr><td>
+ <blockquote class="sectionBody">
+ <table class="subsectionTable">
+ <tr><td>
+
+
+
+ <a name="Preliminary">
+ <h2>Preliminary
+ </h2>
+ </a>
+ </td></tr>
+ <tr><td>
+ <blockquote class="subsectionBody">
+ <p>
+ This page collects information on running DUCC
+ in Cloud environments.
+ </p>
+ <p>It is preliminary, and is missing details.
+ Nevertheless, it may have some potentially useful references and ideas.
+ More will be added as it becomes available.</p>
+ </blockquote>
+ </td></tr>
+ </table>
+ <table class="subsectionTable">
+ <tr><td>
+
+
+
+ <a name="External Links">
+ <h2>External Links
+ </h2>
+ </a>
+ </td></tr>
+ <tr><td>
+ <blockquote class="subsectionBody">
+ <ul>
+ <li><p>
+ <a target="_blank" rel="nofollow noopener" href="https://github.com/aleksey-hariton/uima-ducc-docker">
+ https://github.com/aleksey-hariton/uima-ducc-docker</a>
+ is a github repository with one approach.
+ </p></li>
+ </ul>
+ </blockquote>
+ </td></tr>
+ </table>
+ <table class="subsectionTable">
+ <tr><td>
+
+
+
+ <a name="Considerations">
+ <h2>Considerations
+ </h2>
+ </a>
+ </td></tr>
+ <tr><td>
+ <blockquote class="subsectionBody">
+ <p style="margin-left: 2rem">NOTE: This information is from an email to the uima-dev list by one of the DUCC committers.
+ In this email, they reference a "glusterfs" which is a shared file system cross mounted
+ on various machines. This "shared file system" approach is the conventional way DUCC is
+ configured, but it can be configured without a shared file system; see the DUCC docs.</p>
+ <p>We've been running DUCC in a cloud environment for almost a year now. The
+DUCC master and a
+<a target="_blank" rel="nofollow noopener" href="https://www.google.com/search?q=glusterfs">glusterfs</a>
+servers run on bare metal and all of the
+workstations and worker machines run on VMs. Cluster users add VMs to the
+cluster as needed. A job can be started on one more workers and then
+additional VMs dynamically added to which the job will automatically scale
+out to use. A common system image is maintained on all VM machines via an
+LDAP server and shared filesystem data. Users belong to groups and share
+machines allocated by members of the group.
+</p>
+ <p>
+A DUCC VM-image is used to automatically connect new VMs to the DUCC master
+and glusterfs. The DUCC master configuration may be updated anytime, for
+example to add new groups or even update master software. VMs automatically
+sync DUCC software and configuration each time they start their DUCC agent.
+The VM image supports three different machine types: a graphical
+workstation, a CPU worker and a GPU worker (used typically for deep learning
+training). DUCC spawns work on specified
+worker machine types and even specific machines. Workstations are optional
+as DUCC requests can be submitted from worker machines. Docker images are
+supported using <a target="_blank" rel="nofollow noopener" href="https://www.google.com/search?q=podman">Podman</a>. Podman runs rootless and only allows access to all
+mounted file systems with user credentials.
+</p>
+ <p>
+
+In order to keep some level of data security, a group directory is only
+mounted on the VMs created by members of the group. Individual users
+maintain file permissions as desired, but, as anyone that creates a VM has
+root access, they could become any other user and access data from other
+group members.There is a self-service glusterfs webapp that is used to
+export group data to new VMs and manage quotas.
+</p>
+ <p>
+
+The VM-image builder and glusterfs webapp are not yet part of Apache DUCC.
+
+</p>
+ </blockquote>
+ </td></tr>
+ </table>
+ </blockquote>
+ </p>
+ </td></tr>
+ </table>
+ </td>
+ </tr>
+ <!-- FOOTER -->
+ <tr><td colspan="2">
+ <hr noshade="" size="1"/>
+ </td></tr>
+ <tr><td colspan="2">
+ <table class="pageFooter">
+ <tr>
+ <td><a href="index.html">Home</a></td>
+ <td><a href="privacy-policy.html">Privacy Policy</a></td>
+ <td style="font-size:75%">
+ Copyright © 2006-2013, The Apache Software Foundation.<br/>
+ Apache UIMA, UIMA, the Apache UIMA logo and the Apache Feather logo are trademarks of The Apache Software Foundation.<br/>
+ All other marks mentioned may be trademarks or registered trademarks of their respective owners.
+ </td>
+ <td><a href="mailto:dev@uima.apache.org">Contact us</a></td>
+ </tr>
+ </table>
+ </td></tr>
+ </table>
+ </body>
+ </html>
+
Modified: uima/site/trunk/uima-website/xdocs/doc-uimaducc-whatitam.xml
URL: http://svn.apache.org/viewvc/uima/site/trunk/uima-website/xdocs/doc-uimaducc-whatitam.xml?rev=1875373&r1=1875372&r2=1875373&view=diff
==============================================================================
--- uima/site/trunk/uima-website/xdocs/doc-uimaducc-whatitam.xml (original)
+++ uima/site/trunk/uima-website/xdocs/doc-uimaducc-whatitam.xml Wed Mar 18 17:27:33 2020
@@ -40,7 +40,7 @@ This guide is intended to be a user orie
DUCC is a Linux cluster controller designed to scale out any UIMA pipeline
for high throughput collection processing jobs as well as for low latency
real-tme applications.
-Building on UIMA-AS, DUCC is particularly well suited to run large memory Java
+DUCC is particularly well suited to run large memory Java
analytics in multiple threads in order to fully utilize multicore machines.
DUCC manages the life cycle of all processes deployed across the cluster, including
non-UIMA processes such as tomcat servers or VNC sessions.
@@ -224,6 +224,15 @@ by adding a single parameter to the Job
<p>
</p>
</subsection>
+
+ <subsection name="DUCC on Clouds">
+ <p>
+ DUCC has been running in various cloud deployments.
+ The documentation for this is still in-progress, but some information is
+ available <a target="_blank" rel="nofollow noopener"
+ href="doc-ducc-cloud">here</a>.
+ </p>
+ </subsection>
<subsection name="DUCC - What next?">
<p>
Added: uima/site/trunk/uima-website/xdocs/ducc-cloud.xml
URL: http://svn.apache.org/viewvc/uima/site/trunk/uima-website/xdocs/ducc-cloud.xml?rev=1875373&view=auto
==============================================================================
--- uima/site/trunk/uima-website/xdocs/ducc-cloud.xml (added)
+++ uima/site/trunk/uima-website/xdocs/ducc-cloud.xml Wed Mar 18 17:27:33 2020
@@ -0,0 +1,107 @@
+<?xml version="1.0" encoding="ISO-8859-1"?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<document>
+
+ <properties>
+ <title>Running UIMA DUCC in a Cloud environment</title>
+ <author email="dev@uima.apache.org">
+ Apache UIMA Team
+ </author>
+ </properties>
+
+ <body>
+ <section
+ name="DUCC in Clouds">
+ <subsection name="Preliminary">
+ <p>
+ This page collects information on running DUCC
+ in Cloud environments.
+ </p>
+
+ <p>It is preliminary, and is missing details.
+ Nevertheless, it may have some potentially useful references and ideas.
+ More will be added as it becomes available.</p>
+ </subsection>
+
+ <subsection name="External Links">
+ <ul>
+ <li><p>
+ <a target="_blank" rel="nofollow noopener"
+ href="https://github.com/aleksey-hariton/uima-ducc-docker">
+ https://github.com/aleksey-hariton/uima-ducc-docker</a>
+ is a github repository with one approach.
+ </p></li>
+ </ul>
+ </subsection>
+
+ <subsection name="Considerations">
+ <p style="margin-left: 2rem">NOTE: This information is from an email to the uima-dev list by one of the DUCC committers.
+ In this email, they reference a "glusterfs" which is a shared file system cross mounted
+ on various machines. This "shared file system" approach is the conventional way DUCC is
+ configured, but it can be configured without a shared file system; see the DUCC docs.</p>
+
+ <p>We've been running DUCC in a cloud environment for almost a year now. The
+DUCC master and a
+<a target="_blank" rel="nofollow noopener"
+href="https://www.google.com/search?q=glusterfs">glusterfs</a>
+servers run on bare metal and all of the
+workstations and worker machines run on VMs. Cluster users add VMs to the
+cluster as needed. A job can be started on one more workers and then
+additional VMs dynamically added to which the job will automatically scale
+out to use. A common system image is maintained on all VM machines via an
+LDAP server and shared filesystem data. Users belong to groups and share
+machines allocated by members of the group.
+</p>
+<p>
+A DUCC VM-image is used to automatically connect new VMs to the DUCC master
+and glusterfs. The DUCC master configuration may be updated anytime, for
+example to add new groups or even update master software. VMs automatically
+sync DUCC software and configuration each time they start their DUCC agent.
+The VM image supports three different machine types: a graphical
+workstation, a CPU worker and a GPU worker (used typically for deep learning
+training). DUCC spawns work on specified
+worker machine types and even specific machines. Workstations are optional
+as DUCC requests can be submitted from worker machines. Docker images are
+supported using <a target="_blank" rel="nofollow noopener"
+href="https://www.google.com/search?q=podman">Podman</a>. Podman runs rootless and only allows access to all
+mounted file systems with user credentials.
+</p>
+<p>
+
+In order to keep some level of data security, a group directory is only
+mounted on the VMs created by members of the group. Individual users
+maintain file permissions as desired, but, as anyone that creates a VM has
+root access, they could become any other user and access data from other
+group members.There is a self-service glusterfs webapp that is used to
+export group data to new VMs and manage quotas.
+</p>
+<p>
+
+The VM-image builder and glusterfs webapp are not yet part of Apache DUCC.
+
+</p>
+ </subsection>
+ </section>
+ </body>
+
+</document>
+