You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Kris Nuttycombe <Kr...@noaa.gov> on 2005/02/04 17:34:29 UTC
Re: RFI : jakarta commons - Pipeline Component
Hi, Abhilash,
I'm the main developer of the pipeline, so I can help you out. The
javadocs have been broken for some time and I haven't been able to get
anyone with permission to update the website to fix the link, so I've
attached a tarball of the docs to this email.
The source code is available from both CVS and Subversion (there are no
releases as yet.) You can get access starting here:
http://jakarta.apache.org/site/cvsindex.html
Hope this helps! Any comments or contributions are greatly appreciated!
Kris
Abhilash Koneri wrote:
>Hi,
>
>I came accross the pipeline component on the webpage at jakarta -
>http://jakarta.apache.org/commons/sandbox/pipeline/ . However, I was
>not able to find the downloads or the java doc pages. I am very
>interesting in this compnent. I am a java developer and would like to
>help if possible. Could you point me to some documentation / source
>code for this component?
>
>Thanks much.
>
>Abhilash Koneri
>(http://www.angelfire.com/or/abhilash/site/index.html)
>
>
>
--
=====================================================
Kris Nuttycombe
Associate Scientist
Geospatial Data Services Group
CIRES, National Geophysical Data Center/NOAA
(303) 497-6337
Kris.Nuttycombe@noaa.gov
=====================================================
[pipeline] To-do list
Posted by Kris Nuttycombe <Kr...@noaa.gov>.
Hi, Abhilash,
Thanks! I'm sorry that it's taken me so long to get back to you on this
. I believe that Craig had generated the build.xml file from the maven
project file (since he uses ant for the nightlies) and had just
committed whatever was generated for him.
As far as a to-do list goes, I'm currently working on trying to
implement a set of graceful failure modes for pipeline stages. I've
toyed around with a few different ideas on how to go about doing this,
but I'd be grateful for any input you might have. A few failure cases
that I need to solve are listed below:
1) Failure of a downstream dependency: If the preprocess() method of a
given stage fails, cause one or more upstream stages to terminate execution.
2) Failure of an upstream stage where a downstream stage is blocked
waiting on an event that the upstream stage might generate. It's
currently possible to deadlock the pipeline if this happens, so I need
to come up with some set of configurable conditions under which it's
allowable to interrupt the blocked stage.
3) Stateful suspend and resume. This is a big task that needs a lot of
thought - how to serialize the pipeline state so that it's possible to
resume processing after a failure. Obviously whatever decisions are made
here will make a difference in how stages are implemented, so we may
simply want to specify a "Resumable" interface that the stages can
optionally implement and leave it up to the implementer to handle
suspension and resumption. At very least, we should probably provide
utility methods for serializing and deserializing a stage's queue.
Another concept that I've started playing around with is a "router"
stage that can route data onto different pipeline branches using a
chain-of-responsibility for the decision-making process.
Kris
Abhilash Koneri wrote:
>Kris,
>
>Thanks for the info. I downloaded the source and compiled it. I have a
>fair idea now on the source and would be interested in contributing.
>Do you have a to-do list for further development?
>
>The build.xml was not generic enough. It had reference to specific
>directories that did not enable anyone to build out-of-the-box (ex.
>/home/craigmcc/). I have cleaned it up a bit. Also, I added a
>maven-get task to obtain a dependent library -
><get dest="${libdir}/commons-collections-3.1.jar" usetimestamp="true"
>ignoreerrors="true"
>src="http://www.ibiblio.org/maven/commons-collections/jars/commons-collections-3.1.jar">
>
>
>Attached is the version I now use. Hope it helps.
>
>Abhilash
>
>
>On Fri, 04 Feb 2005 09:34:29 -0700, Kris Nuttycombe
><Kr...@noaa.gov> wrote:
>
>
>>Hi, Abhilash,
>>
>>I'm the main developer of the pipeline, so I can help you out. The
>>javadocs have been broken for some time and I haven't been able to get
>>anyone with permission to update the website to fix the link, so I've
>>attached a tarball of the docs to this email.
>>
>>The source code is available from both CVS and Subversion (there are no
>>releases as yet.) You can get access starting here:
>>http://jakarta.apache.org/site/cvsindex.html
>>
>>Hope this helps! Any comments or contributions are greatly appreciated!
>>
>>Kris
>>
>>
>>Abhilash Koneri wrote:
>>
>>
>>
>>>Hi,
>>>
>>>I came accross the pipeline component on the webpage at jakarta -
>>>http://jakarta.apache.org/commons/sandbox/pipeline/ . However, I was
>>>not able to find the downloads or the java doc pages. I am very
>>>interesting in this compnent. I am a java developer and would like to
>>>help if possible. Could you point me to some documentation / source
>>>code for this component?
>>>
>>>Thanks much.
>>>
>>>Abhilash Koneri
>>>(http://www.angelfire.com/or/abhilash/site/index.html)
>>>
>>>
>>>
>>>
>>>
>>--
>>=====================================================
>>Kris Nuttycombe
>>Associate Scientist
>>Geospatial Data Services Group
>>CIRES, National Geophysical Data Center/NOAA
>>(303) 497-6337
>>Kris.Nuttycombe@noaa.gov
>>=====================================================
>>
>>
>>
>>
>>
>
>
>
>
>------------------------------------------------------------------------
>
><?xml version="1.0" encoding="UTF-8"?>
>
><!--build.xml generated by maven from project.xml version 0.0.1
> on date October 2 2004, time 1145-->
>
><project default="jar" name="commons-pipeline" basedir=".">
> <property name="defaulttargetdir" value="target">
> </property>
> <property name="libdir" value="${defaulttargetdir}/lib">
> </property>
> <property name="classesdir" value="${defaulttargetdir}/classes">
> </property>
> <property name="testclassesdir" value="${defaulttargetdir}/test-classes">
> </property>
> <property name="distdir" value="dist">
> </property>
> <property name="javadocdir" value="dist/docs/api">
> </property>
> <property name="final.name" value="commons-pipeline-0.0.1">
> </property>
> <target name="init" description="o Initializes some properties">
> <mkdir dir="${libdir}">
> </mkdir>
> <condition property="noget">
> <equals arg2="only" arg1="${build.sysclasspath}">
> </equals>
> </condition>
> </target>
> <target name="compile" description="o Compile the code" depends="get-deps">
> <mkdir dir="${classesdir}">
> </mkdir>
> <javac destdir="${classesdir}" deprecation="true" debug="true" optimize="false" excludes="**/package.html">
> <src>
> <pathelement location="src/java">
> </pathelement>
> </src>
> <classpath>
> <fileset dir="${libdir}">
> <include name="*.jar">
> </include>
> </fileset>
> </classpath>
> </javac>
> </target>
> <target name="jar" description="o Create the jar" depends="compile,test">
> <jar jarfile="${defaulttargetdir}/${final.name}.jar" excludes="**/package.html" basedir="${classesdir}">
> </jar>
> </target>
> <target name="clean" description="o Clean up the generated directories">
> <delete dir="${defaulttargetdir}">
> </delete>
> <delete dir="${distdir}">
> </delete>
> </target>
> <target name="dist" description="o Create a distribution" depends="jar, javadoc">
> <mkdir dir="dist">
> </mkdir>
> <copy todir="dist">
> <fileset dir="${defaulttargetdir}" includes="*.jar">
> </fileset>
> <fileset dir="${basedir}" includes="LICENSE*, README*">
> </fileset>
> </copy>
> </target>
> <target name="test" description="o Run the test cases" if="test.failure" depends="internal-test">
> <fail message="There were test failures.">
> </fail>
> </target>
> <target name="internal-test" depends="compile-tests">
> </target>
> <target name="compile-tests" depends="compile">
> </target>
> <target name="javadoc" description="o Generate javadoc">
> <mkdir dir="${javadocdir}">
> </mkdir>
> <tstamp>
> <format pattern="2004-yyyy" property="year">
> </format>
> </tstamp>
> <property name="copyright" value="Copyright &copy; The Apache Software Foundation. All Rights Reserved.">
> </property>
> <property name="title" value="commons-pipeline 0.0.1 API">
> </property>
> <javadoc use="true" private="true" destdir="${javadocdir}" author="true" version="true" sourcepath="src/java" packagenames="org.apache.commons.pipeline.*">
> <classpath>
> <fileset dir="${libdir}">
> <include name="*.jar">
> </include>
> </fileset>
> <pathelement location="${defaulttargetdir}/${final.name}.jar">
> </pathelement>
> </classpath>
> </javadoc>
> </target>
> <target name="get-deps" unless="noget" depends="init">
> <get dest="${libdir}/commons-digester-1.6.jar" usetimestamp="true" ignoreerrors="true" src="http://www.ibiblio.org/maven/commons-digester/jars/commons-digester-1.6.jar">
> </get>
> <get dest="${libdir}/commons-net-1.2.1.jar" usetimestamp="true" ignoreerrors="true" src="http://www.ibiblio.org/maven/commons-net/jars/commons-net-1.2.1.jar">
> </get>
> <get dest="${libdir}/log4j-1.2.8.jar" usetimestamp="true" ignoreerrors="true" src="http://www.ibiblio.org/maven/log4j/jars/log4j-1.2.8.jar">
> </get>
> <get dest="${libdir}/junit-3.8.1.jar" usetimestamp="true" ignoreerrors="true" src="http://www.ibiblio.org/maven/junit/jars/junit-3.8.1.jar">
> </get>
> <get dest="${libdir}/ant-1.5.jar" usetimestamp="true" ignoreerrors="true" src="http://www.ibiblio.org/maven/ant/jars/ant-1.5.jar">
> </get>
> <get dest="${libdir}/ant-optional-1.5.jar" usetimestamp="true" ignoreerrors="true" src="http://www.ibiblio.org/maven/ant/jars/ant-optional-1.5.jar">
> </get>
> <get dest="${libdir}/commons-collections-3.1.jar" usetimestamp="true" ignoreerrors="true" src="http://www.ibiblio.org/maven/commons-collections/jars/commons-collections-3.1.jar">
> </get>
> </target>
> <target name="install-maven">
> <get dest="${user.home}/maven-install-latest.jar" usetimestamp="true" src="${repo}/maven/maven-install-latest.jar">
> </get>
> <unjar dest="${maven.home}" src="${user.home}/maven-install-latest.jar">
> </unjar>
> </target>
></project
>
--
=====================================================
Kris Nuttycombe
Associate Scientist
Geospatial Data Services Group
CIRES, National Geophysical Data Center/NOAA
(303) 497-6337
Kris.Nuttycombe@noaa.gov
=====================================================
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org