You are viewing a plain text version of this content. The canonical link for it is here.
Posted to s4-commits@incubator.apache.org by mm...@apache.org on 2012/07/26 10:52:41 UTC
[6/16] git commit: Add release information files,
update scripts to generate release artefacts - also cleaned files in
twitter example
Add release information files, update scripts to generate release artefacts
- also cleaned files in twitter example
Project: http://git-wip-us.apache.org/repos/asf/incubator-s4/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-s4/commit/94f54a19
Tree: http://git-wip-us.apache.org/repos/asf/incubator-s4/tree/94f54a19
Diff: http://git-wip-us.apache.org/repos/asf/incubator-s4/diff/94f54a19
Branch: refs/heads/piper
Commit: 94f54a19df0bd2145b0d567d1a9bf74cbcc85785
Parents: e679cdb
Author: Matthieu Morel <mm...@apache.org>
Authored: Wed Jul 25 17:46:53 2012 +0200
Committer: Matthieu Morel <mm...@apache.org>
Committed: Wed Jul 25 18:26:22 2012 +0200
----------------------------------------------------------------------
DISCLAIMER | 15 +
LICENSE | 204 +++++++++++++++
NOTES.md | 101 -------
NOTICE.txt | 16 +-
README.md | 132 ++++++----
RELEASE_NOTES.html | 118 +++++++++
build.gradle | 132 ++++++++--
etc/s4-counter-example.png | Bin 60513 -> 0 bytes
etc/s4-objects-example.png | Bin 19223 -> 0 bytes
s4 | 10 -
.../org/apache/s4/core/ft/StorageCallback.java | 10 +-
.../s4/core/window/AbstractSlidingWindowPE.java | 13 +-
subprojects/s4-tools/s4-tools.gradle | 24 +-
.../src/main/resources/s4.properties | 18 --
test-apps/twitter-counter/README.txt | 22 +--
.../src/main/resources/default.s4.properties | 19 --
16 files changed, 552 insertions(+), 282 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/DISCLAIMER
----------------------------------------------------------------------
diff --git a/DISCLAIMER b/DISCLAIMER
new file mode 100644
index 0000000..789090b
--- /dev/null
+++ b/DISCLAIMER
@@ -0,0 +1,15 @@
+Apache S4 is an effort undergoing incubation at the Apache Software
+Foundation (ASF), sponsored by the Apache Incubator PMC.
+
+Incubation is required of all newly accepted projects until a further review
+indicates that the infrastructure, communications, and decision making process
+have stabilized in a manner consistent with other successful ASF projects.
+
+While incubation status is not necessarily a reflection of the completeness
+or stability of the code, it does indicate that the project has yet to be
+fully endorsed by the ASF.
+
+For more information about the incubation status of the S4 project you
+can go to the following page:
+
+http://incubator.apache.org/s4
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/LICENSE
----------------------------------------------------------------------
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..f10b48c
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,204 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/NOTES.md
----------------------------------------------------------------------
diff --git a/NOTES.md b/NOTES.md
deleted file mode 100644
index c096e43..0000000
--- a/NOTES.md
+++ /dev/null
@@ -1,101 +0,0 @@
-# Manually packaging an S4 app.
-
-The app should extend io.s4.core.App:
-
-<pre>
-import io.s4.core.App;
-
-
-public class HelloApp extends App {
-
- @Override
- protected void start() {
- System.out.println("Hello App! I'm starting...");
- }
-
- @Override
- protected void init() {
- System.out.println("Hello App! I'm initing...");
-
- }
-
- @Override
- protected void close() {
- System.out.println("Hello App! I'm closing...");
-
- }
-}
-</pre>
-
-We want to indicate the entry point in the jar. To do this I create a manifest file.
-
-* Edit MANIFEST.MF:
-
-<pre>
-Manifest-Version: 1.0
-provider: gradle
-Implementation-Url: http://incubator.apache.org/projects/s4.html
-Implementation-Version: 0.5.0-SNAPSHOT
-Implementation-Vendor: Apache S4
-Implementation-Vendor-Id: io.s4
-S4-App-Class: HelloApp
-</pre>
-
-We will use the attribute prefix "S4-App-" for all S3-related properties.
-
-* Files:
- HelloApp.class
- HelloApp.java
- MANIFEST.MF
-
-* Create the jar file:
- jar cmf MANIFEST.MF HelloApp.jar HelloApp.class
-
-* Check:
- jar tvf HelloApp.jar
- 0 Mon Oct 17 16:31:18 PDT 2011 META-INF/
- 275 Mon Oct 17 16:31:18 PDT 2011 META-INF/MANIFEST.MF
- 702 Mon Oct 17 13:36:50 PDT 2011 HelloApp.class
-
-## Create Counter Example JAR
-
-Copy counter example classfiles:
-
-<pre>
-AgeKeyFinder.class CountKeyFinder.class GenderKeyFinder.class Module.class PrintPE.class UserEvent.class
-CountEvent.class CounterPE.class GenerateUserEventPE.class MyApp.class README.md UserIDKeyFinder.class
-
-$ cat MANIFEST.MF
-Manifest-Version: 1.0
-provider: gradle
-Implementation-Url: http://incubator.apache.org/projects/s4.html
-Implementation-Version: 0.5.0-SNAPSHOT
-Implementation-Vendor: Apache S4
-Implementation-Vendor-Id: io.s4
-S4-App-Class: io.s4.example.counter.MyApp
-
-$ jar cmf MANIFEST.MF CounterExample.s4r org
-$ jar tfv CounterExample.s4r
- 0 Sat Oct 22 19:21:06 PDT 2011 META-INF/
- 307 Sat Oct 22 19:21:06 PDT 2011 META-INF/MANIFEST.MF
- 0 Wed Oct 19 08:55:18 PDT 2011 org/
- 0 Wed Oct 19 08:55:18 PDT 2011 org/apache/
- 0 Wed Oct 19 08:55:36 PDT 2011 org/apache/s4/
- 0 Wed Oct 19 08:55:44 PDT 2011 org/apache/s4/example/
- 0 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/
- 1240 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/AgeKeyFinder.class
- 1708 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/CounterPE.class
- 1330 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/CountEvent.class
- 1197 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/CountKeyFinder.class
- 1254 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/GenderKeyFinder.class
- 2378 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/GenerateUserEventPE.class
- 3414 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/Module.class
- 4196 Sat Oct 22 19:19:14 PDT 2011 org/apache/s4/example/counter/MyApp.class
- 897 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/PrintPE.class
- 532 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/README.md
- 1233 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/UserEvent.class
- 1198 Wed Oct 19 08:55:18 PDT 2011 org/apache/s4/example/counter/UserIDKeyFinder.class
-
-</pre>
-
-
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/NOTICE.txt
----------------------------------------------------------------------
diff --git a/NOTICE.txt b/NOTICE.txt
index b0a385f..578e5d5 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -1,13 +1,5 @@
-=======================================================================
-NOTICE file for use with, and corresponding to Section 4 of,
-the Apache License, Version 2.0, in this case for the S4 project.
-=========================================================================
+Apache S4
+Copyright 2012 The Apache Software Foundation
- This product includes software developed by
- Yahoo! Inc. (www.yahoo.com)
- Copyright (c) 2010, 2011 Yahoo! Inc. All rights reserved.
-
- This product includes software developed by
- The S4 Project community (http://s4.io)
- Copyright (c) 2011 The S4 Project All rights reserved.
-
\ No newline at end of file
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 931ddbc..4e382b7 100644
--- a/README.md
+++ b/README.md
@@ -1,52 +1,84 @@
-S4-Piper Design
-===============
-
-The goal of this project is to incorporate feedback from various users and create a prototype with a cleaner, simpler API.
-The base classes are the foundation on top of which higher level layers and tools can be built. For example, one could write
-a domain specific language to create the application graph. Look at this as the foundation and not as the end product.
-
-Here is a list of ideas:
-
-- Eliminate string identifiers completely and make it work nicely with Guice.
-- Take advantage of Java strong typing using generics in the framework. (But keep it simple for the app developer.)
-- Use dependency injection, no more "new".
-- Limit property files to simple parameters. Config file should not be
-a programming language. (No more XML.)
-- Configure the app in a GuiceModule.
-- Make fields final private whenever possible and use constructors
-instead of setter methods.
-- Make PEs as immutable as possible.
-- Hide details about partitioning from app developer. The idea is to
-create a graph for the prototypes and have the base classes deal with
-the distributed processing details.
-- Use osgi to create s4 app bundles that can be dropped in a directory
-and get loaded by s4. (This will take a bit longer to do.)
-
-Please post feedback and suggestions in the s4 forum.
-
-The repo is here:
-
-https://github.com/leoneu/s4-piper
-
-In this version:
-
-- Everything is a POJO.
-- Key framework classes (io.s4) are:
- - Event: all messages sent between PEs inherit from Event.
- - ProcessingElement: base class for PEs.
- - Stream: takes an Event subclass as the parameter type for the stream. A stream holds references to the target PEs. Streams are the edges between PEs.
- - Key: is a helper class that can return the value of a key for a specific type of event. For each type of event, one will create a KeyFinder class that knows how to get the value from the Event class.
- - App: the container for the app.
-
-Note that when we create a PE in App, we are really creating a prototype of the PE. The actual PE instances get created every time a node receives an event that is dispatched to a specific PE prototype. The first PE has Map that holds the instances by a specific key. PE instances are clones which means that we don't call the constructor. Instead we provide an init method. Also, the instance variables are cloned. This means that references are also copied. If you wanted to have a List per PE instance, you would have to create it in the init method. This design pattern, makes thing very simple and efficient but has the downside that programmers have to understand how instances are created via clone.
-
-For now we identify PEs using a composite string produced by KeyFinder. We must also pass the raw data types used to create the composite key. Need to think how to do this with type safety in mind.
-
-
-Examples:
-
-* io.s4.example.counter - Generates dummy events that are counted using various keys.
-* io.s4.example.model - Training and testing a probabilistic classifier.
-
+Apache S4
+=========
+
+Overview
+--------
+S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.
+
+S4 0.5.0 is a complete refactoring of the previous version of S4. It grounds on the same concepts (partitioning inspired by map-reduce, actors-like distribution model), but with the following objectives:
+
+- cleaner and simpler API
+- robust configuration through statically defined modules
+- cleaner architecture
+- robust codebase
+- easier to develop S4 apps, to test, and to use the platform
+
+We added the following core features:
+
+- TCP-based communications
+- state recovery through a flexible checkpointing mechanism
+- inter-cluster/app communications through a pub-sub model
+- dynamic application deployment
+- toolset for creating nodes, adding
+
+
+Documentation
+-------------
+
+For the latest information about S4, please visit our website at:
+
+ http://inbubator.apache.org/s4
+
+and our wiki, at:
+
+ https://cwiki.apache.org/confluence/display/S4/S4+Wiki
+
+Currently the wiki contains the most relevant and up-to-date documentation.
+
+Source code is available here: https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git
+
+
+
+How to build
+------------
+This only applies if you checkout from the source repository or if you download a released source package.
+
+You need a 6+ JDK.
+
+We use gradle http://gradle.org as the build system.
+
+* From the root directory of the S4 project:
+
+./gradlew install
+
+This will build the packages and install the artefacts in the local maven repository.
+
+* Then, build the tools:
+
+./gradlew s4-tools:installApp
+
+This will build the tools so that you can drive the platform through the "s4" command.
+
+
+Directory structure
+-------------------
+* If you have a source package:
+ - root directory contains build and utility scripts, as well as informative files
+ - config/ directory contains configuration files for source code formatting
+ - doc/ directory contains the javadoc
+ - gradle/ directory contains libraries used by the gradle build tool
+ - lib/ directory contains some other gradle libraries
+ - subproject/ directory contains the plaftorm subprojects: base, comm, core, tools, as well as example (example is not fully ported to 0.5.0)
+ - test-apps/ directory contains some examples (some of them very trivial are used in regression tests)
+
+
+
+* If you have a binary package:
+ - root directory contains the s4 utility script as well as informative files
+ - doc/ directory contains the javadoc
+ - lib/ directory contains :
+ * the platform libraries (prefixed with "s4")
+ * the dependencies
+
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/RELEASE_NOTES.html
----------------------------------------------------------------------
diff --git a/RELEASE_NOTES.html b/RELEASE_NOTES.html
new file mode 100644
index 0000000..4c1a67b
--- /dev/null
+++ b/RELEASE_NOTES.html
@@ -0,0 +1,118 @@
+
+<p>
+ Release Notes - Apache S4 - Version 0.5.0
+</p>
+<p>
+S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform
+that allows programmers to easily develop applications for processing continuous unbounded streams of data.
+</p>
+<p>
+Apache S4 0.5.0 is the first release of S4 as an Apache Incubator project.
+It is also a complete refactoring of the previous S4 codebase.
+</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-24'>S4-24</a>] - Dynamic deployment of S4 applications from a remote repository
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-29'>S4-29</a>] - Remove file-based configurations
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-33'>S4-33</a>] - TestUtils and modules should be split across s4-comm and s4-core
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-36'>S4-36</a>] - CommTests should use ZkBasedClusterManagement
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-65'>S4-65</a>] - Create s4 status command
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-12'>S4-12</a>] - Dynamic Method Dispatching
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-14'>S4-14</a>] - Listener.recv() behavior when interrupted
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-18'>S4-18</a>] - gradle test fails
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-23'>S4-23</a>] - NettyEmitter -- 1. Infinitely waiting on connection, 2. numPartitions is same as numClusterNodes
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-26'>S4-26</a>] - standalone application build script fails when the source directory contains more than 1 class
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-43'>S4-43</a>] - Replace deprecated method in Gradle
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-56'>S4-56</a>] - Prevent concurrent access to published s4r files
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-61'>S4-61</a>] - Threads running the PEs don't have a Context Class Loader
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-63'>S4-63</a>] - onTime() events are not synchronized correctly
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-70'>S4-70</a>] - s4 node command ignores -zk parameter
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-71'>S4-71</a>] - s4 deploy mismanages s4r URI
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-74'>S4-74</a>] - NullPointerException on TCPEmitter.onChange()
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-82'>S4-82</a>] - Fix local communication optimization
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-87'>S4-87</a>] - Checkpointing: recovery : avoid rejections upon fetching
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-6'>S4-6</a>] - Netty initialization
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-17'>S4-17</a>] - Put tmp dir under build
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-21'>S4-21</a>] - Create standard PEs, and Event classes
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-22'>S4-22</a>] - Adaptor + inter app communication
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-27'>S4-27</a>] - extensions to cluster configuration through Zookeeper
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-34'>S4-34</a>] - Rename s4.comm.netty and constituents to s4.comm.tcp
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-44'>S4-44</a>] - optional backoff upon multiple consecutive failed checkpoint fetches
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-57'>S4-57</a>] - Document and improve the windowing PE
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-60'>S4-60</a>] - Development mode for rapid application development and testing
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-67'>S4-67</a>] - S4R packaging: improve packaging of dependencies
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-69'>S4-69</a>] - Update gradle to 1.0
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-72'>S4-72</a>] - Resource loading from S4R
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-75'>S4-75</a>] - Code improvements in comm layer
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-78'>S4-78</a>] - Gradle tasks don't finish as soon as possible
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-81'>S4-81</a>] - S4 tools syntax: remove curly braces + expose custom modules config
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-85'>S4-85</a>] - Improve handling of ZK connection changes
+</li>
+</ul>
+
+<h2> New Feature
+</h2>
+<ul>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-4'>S4-4</a>] - Dynamic App Loading
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-11'>S4-11</a>] - add checkpointing mechanism to s4-piper
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-38'>S4-38</a>] - S4 command line utility
+</li>
+</ul>
+
+<h2> Task
+</h2>
+<ul>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-31'>S4-31</a>] - Facilities for packaging example and test apps
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/S4-88'>S4-88</a>] - Prepare 0.5.0 release
+</li>
+</ul>
+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/build.gradle
----------------------------------------------------------------------
diff --git a/build.gradle b/build.gradle
index db62a11..006a425 100644
--- a/build.gradle
+++ b/build.gradle
@@ -24,13 +24,13 @@
description = 'Apache S4'
defaultTasks 'assemble'
-project.ext["archivesBaseName"] = 's4'
+project.ext["archivesBaseName"] = 'apache-s4'
project.ext["distRootFolder"] = "$archivesBaseName-${-> version}"
allprojects {
// Append the suffix 'SNAPSHOT' when the build is not for release.
- version = new Version(major: 0, minor: 5, bugfix: 0, isRelease: false)
+ version = new Version(major: 0, minor: 5, bugfix: 0, isRelease: true)
group = 'org.apache.s4'
/* Search these repos to find artifacts. Gradle will download and cache. */
@@ -56,7 +56,6 @@ project.ext["libraries"] = [
mockito_core: 'org.mockito:mockito-core:1.9.0',
commons_config: 'commons-configuration:commons-configuration:1.6',
commons_codec: 'commons-codec:commons-codec:1.4',
- commons_httpclient: 'commons-httpclient:commons-httpclient:3.1',
commons_coll: 'net.sourceforge.collections:collections-generic:4.01', // Use this lib until the commons collection with Generics is released.
slf4j: 'org.slf4j:slf4j-api:1.6.1',
logback_core: 'ch.qos.logback:logback-core:0.9.29',
@@ -115,13 +114,22 @@ subprojects {
/* Testing. */
testCompile( libraries.junit )
}
+
+ jar {
+ from (rootDir) {
+ include "NOTICE.txt"
+ include "LICENSE"
+ into "META-INF"
+ }
+ }
manifest.mainAttributes(
- provider: 'gradle',
- 'Implementation-Url': 'http://incubator.apache.org/projects/s4.html',
- 'Implementation-Version': version,
- 'Implementation-Vendor': 'Apache S4',
- 'Implementation-Vendor-Id': 'io.s4'
+ provider: 'gradle',
+ 'Implementation-Url': 'http://incubator.apache.org/s4/',
+ 'Implementation-Version': version,
+ 'Implementation-Vendor': 'Apache Software Foundation',
+ 'Implementation-Vendor-Id': 'org.apache',
+ 'Implementation-Title': 'org.apache.s4.'+project.name.substring(3) // s4-comm becomes org.apache.s4.comm for instance
)
}
@@ -137,34 +145,106 @@ dependencies {
platformLibs platformProjects
}
-project.ext["binDistImage"] = copySpec {
- platformProjects.collect {proj ->
- into ("platform/lib") {
- from proj.sourceSets.main.resources
- from proj.configurations.runtime.getFiles()
- from proj.configurations.archives.allArtifacts.files
+
+task javadoc (type: Javadoc) {
+ destinationDir = new File(project.rootDir, 'doc/javadoc')
+ destinationDir.deleteDir()
+
+ title = "Apache S4 " + version
+ source platformProjects.collect { project ->
+ project.sourceSets.main.allJava
+ }
+ // Might need a classpath
+ classpath = files(subprojects.collect { project ->
+ project.sourceSets.main.compileClasspath
+ })
+}
+
+// TODO parameterize
+task javadocBinRelease (type: Javadoc) {
+ destinationDir = new File(project.rootDir,'subprojects/s4-tools/src/dist/doc/javadoc')
+ destinationDir.deleteDir()
+ title = "Apache S4 " + version
+ source platformProjects.collect { project ->
+ project.sourceSets.main.allJava
+ }
+ // Might need a classpath
+ classpath = files(subprojects.collect { project ->
+ project.sourceSets.main.compileClasspath
+ })
+}
+
+task copyRequiredFilesToS4ToolsDistDir (type: Copy) {
+ copy {
+ from (rootDir) {
+ include "NOTICE.txt"
+ include "README.md"
+ include "DISCLAIMER"
+ include "RELEASE_NOTES.txt"
+ include "gradlew"
+ include "gradle.bat"
+ include "s4"
+ }
+ into new File(project.rootDir.path+'/subprojects/s4-tools/src/dist/')
+ }
+
+ File s4DistFile = new File(project.rootDir.path+'/subprojects/s4-tools/src/dist/s4')
+ ant.replace(file: s4DistFile, token: "\n# NOTE: \"./gradlew s4-tools:installApp\" will prepare/update the tools subproject and related startup scripts", value: "")
+
+ ant.replace(file: s4DistFile, token: "subprojects/s4-tools/build/install/s4-tools/", value: "")
+
+ ant.chmod(file: s4DistFile, perm: "700")
+
+ copy {
+ from new File(rootDir, 'gradle')
+ into new File(project.rootDir.path+'/subprojects/s4-tools/src/dist/gradle')
+ }
+
+ copy {
+ from (new File(rootDir, 'lib')) {
+ include '*.properties'
+ }
+ into new File(project.rootDir.path+'/subprojects/s4-tools/src/dist/lib')
+ }
+}
+
+
+
+task binDist(dependsOn: ["clean", "s4-tools:clean", "javadocBinRelease", "copyRequiredFilesToS4ToolsDistDir", "s4-tools:cleanDistZip", "s4-tools:distZip"]) {
+ doLast {
+ copy {
+ from (project.rootDir.path+"/subprojects/s4-tools/build/distributions/")
+ into (project.buildDir)
+ exclude('tmp')
}
}
}
-task binImage(type: Copy) {
- description = "Create binary image"
- destinationDir = file("$buildDir/s4-image")
- with binDistImage
+task srcDist(dependsOn: ["clean", "javadoc"], type: Zip) {
+ doFirst {
+ platformProjects.collect { project ->
+ project.tasks.clean.execute()
+ }
+ }
+ classifier='src'
+ appendix='zip'
+ version = project.version
+ baseName=project.ext.archivesBaseName
+ destinationDir = project.buildDir
+ from (project.rootDir)
+ exclude ('build')
+ exclude ('.project')
+ exclude ('.classpath')
+ exclude ('tmp')
+ exclude ('.rat-excludes')
}
task clean << {
delete buildDir
}
-// This task lists all the jars. We can use this pattern to create file descriptors.
-task listJars << {
- description = 'List all the jars in the S4 framework.'
- configurations.platformLibs.each { File file -> println file.name }
-}
-
/* Generates the gradlew scripts.
- http://www.gradle.org/1.0-milestone-3/docs/userguide/gradle_wrapper.html */
+ http://www.gradle.org/docs/1.0/userguide/gradle_wrapper.html */
task wrapper(type: Wrapper) {
gradleVersion = '1.0'
jarFile='lib/gradle-wrapper-1.0.jar'
@@ -177,6 +257,6 @@ class Version {
boolean isRelease
String toString() {
- "$major.$minor.$bugfix${isRelease ? '' : '-SNAPSHOT'}"
+ "$major.$minor.$bugfix${isRelease ? '' : '-SNAPSHOT'}-incubating"
}
}
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/etc/s4-counter-example.png
----------------------------------------------------------------------
diff --git a/etc/s4-counter-example.png b/etc/s4-counter-example.png
deleted file mode 100644
index 7d2a897..0000000
Binary files a/etc/s4-counter-example.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/etc/s4-objects-example.png
----------------------------------------------------------------------
diff --git a/etc/s4-objects-example.png b/etc/s4-objects-example.png
deleted file mode 100644
index dd14499..0000000
Binary files a/etc/s4-objects-example.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/s4
----------------------------------------------------------------------
diff --git a/s4 b/s4
index 3d3e0b4..6471486 100755
--- a/s4
+++ b/s4
@@ -10,15 +10,5 @@ S4_SCRIPT_PATH="$S4_DIR/s4"
subprojects/s4-tools/build/install/s4-tools/bin/s4-tools -s4ScriptPath=$S4_SCRIPT_PATH $@
-# EXAMPLES
-# deploy:
-# ./s4 deploy -appName=twitter-counter -buildFile=test-apps/twitter-counter/build.gradle -cluster=s4-test-cluster
-# ./s4 deploy -appName=twitter-adapter -buildFile=test-apps/twitter-adapter/build.gradle -cluster=s4-adapter-cluster
-# newCluster:
-# # examples:
-#./s4 newCluster -c=s4-test-cluster -flp=11000 -nbTasks=2 ; ./s4 newCluster -c=s4-adapter-cluster -flp=13000 -nbTasks=1
-
-# node:
-# ./s4 node -c=s4-test-cluster
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/subprojects/s4-core/src/main/java/org/apache/s4/core/ft/StorageCallback.java
----------------------------------------------------------------------
diff --git a/subprojects/s4-core/src/main/java/org/apache/s4/core/ft/StorageCallback.java b/subprojects/s4-core/src/main/java/org/apache/s4/core/ft/StorageCallback.java
index 95ea7ab..8e970e3 100644
--- a/subprojects/s4-core/src/main/java/org/apache/s4/core/ft/StorageCallback.java
+++ b/subprojects/s4-core/src/main/java/org/apache/s4/core/ft/StorageCallback.java
@@ -18,18 +18,20 @@
package org.apache.s4.core.ft;
+import org.apache.s4.core.ft.CheckpointingFramework.StorageResultCode;
+
/**
- *
+ *
* Callback for reporting the result of an asynchronous storage operation
- *
+ *
*/
public interface StorageCallback {
/**
* Notifies the result of a storage operation
- *
+ *
* @param resultCode
- * code for the result : {@link SafeKeeper.StorageResultCode}
+ * code for the result : {@link StorageResultCode}
* @param message
* whatever message object is suitable
*/
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/subprojects/s4-core/src/main/java/org/apache/s4/core/window/AbstractSlidingWindowPE.java
----------------------------------------------------------------------
diff --git a/subprojects/s4-core/src/main/java/org/apache/s4/core/window/AbstractSlidingWindowPE.java b/subprojects/s4-core/src/main/java/org/apache/s4/core/window/AbstractSlidingWindowPE.java
index 1edf484..3c3cd67 100644
--- a/subprojects/s4-core/src/main/java/org/apache/s4/core/window/AbstractSlidingWindowPE.java
+++ b/subprojects/s4-core/src/main/java/org/apache/s4/core/window/AbstractSlidingWindowPE.java
@@ -70,8 +70,8 @@ public abstract class AbstractSlidingWindowPE<T extends Slot<U>, U, V> extends P
/**
*
- * Constructor for the event-based slot. The abstract method {@link #addPeriodicSlot()} must be called by the
- * concrete class.
+ * Constructor for the event-based slot. The abstract method {@link #addSlot()} must be called by the concrete
+ * class.
*
* @param app
* the application
@@ -83,7 +83,7 @@ public abstract class AbstractSlidingWindowPE<T extends Slot<U>, U, V> extends P
}
/**
- * Constructor for time-based slots. The abstract method {@link #addPeriodicSlot()} is called periodically.
+ * Constructor for time-based slots. The abstract method {@link #addSlot()} is called periodically.
*
* @param app
* the application
@@ -148,15 +148,12 @@ public abstract class AbstractSlidingWindowPE<T extends Slot<U>, U, V> extends P
* User provided function that evaluates the whole content of the window. It must iterate across all slots. Current
* slots are passed as a parameter and the PE instance is expected to be locked so that iteration over the slots is
* safe.
- *
- * @return
*/
abstract protected V evaluateWindow(Collection<T> slots);
/**
- * Add an object to the sliding window. Use it when the window is not periodic.
- *
- * @param slot
+ * Add a slot to the sliding window. Called automatically for periodic slots. Use it when the window is not
+ * periodic.
*/
protected final void addSlot() {
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/subprojects/s4-tools/s4-tools.gradle
----------------------------------------------------------------------
diff --git a/subprojects/s4-tools/s4-tools.gradle b/subprojects/s4-tools/s4-tools.gradle
index ee1c721..8f47441 100644
--- a/subprojects/s4-tools/s4-tools.gradle
+++ b/subprojects/s4-tools/s4-tools.gradle
@@ -17,7 +17,7 @@
*/
-description = 'The S4 core platform.'
+description = 'The S4 tools platform.'
apply plugin: 'java'
@@ -49,26 +49,20 @@ task copyTemplates << {
}
}
-applicationDistribution.from(configurations.compile) {
- // this is supposed to be done by default but is not anymore in 1.0-rc-3
- include "*.jar"
- into("lib")
-}
-
-startScripts {
- classpath = files('$APP_HOME/lib/*')
-}
-
-
-
-
run {
// run doesn't yet directly accept command line parameters...
if ( project.hasProperty('args') ) {
args project.args.split('\\s+')
print args
}
- }
+ }
+
+distZip {
+ // what we build here is really a binary release of S4 with required subprojects and dependencies (hence refs to rootProject)
+ classifier='bin'
+ version = rootProject.version
+ baseName=rootProject.ext.archivesBaseName
+}
test {
forkEvery=1
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/test-apps/twitter-adapter/src/main/resources/s4.properties
----------------------------------------------------------------------
diff --git a/test-apps/twitter-adapter/src/main/resources/s4.properties b/test-apps/twitter-adapter/src/main/resources/s4.properties
deleted file mode 100644
index fdd1bf6..0000000
--- a/test-apps/twitter-adapter/src/main/resources/s4.properties
+++ /dev/null
@@ -1,18 +0,0 @@
-comm.queue_emmiter_size = 8000
-comm.queue_listener_size = 8000
-cluster.hosts = localhost
-cluster.ports = 5077
-cluster.name = s4-adapter-cluster
-cluster.zk_address = localhost:2181
-cluster.zk_session_timeout = 10000
-cluster.zk_connection_timeout = 10000
-comm.module = org.apache.s4.core.adapter.AdapterModule
-s4.logger_level = DEBUG
-appsDir=/tmp/deploy-test
-tcp.partition.queue_size=1000
-comm.timeout=100
-comm.retry_delay=100
-comm.retries=10
-
-# specify the name of the remote cluster (there is currently only 1 remote cluster max)
-cluster.remote.name=s4-test-cluster
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/test-apps/twitter-counter/README.txt
----------------------------------------------------------------------
diff --git a/test-apps/twitter-counter/README.txt b/test-apps/twitter-counter/README.txt
index d46b491..17eafb9 100644
--- a/test-apps/twitter-counter/README.txt
+++ b/test-apps/twitter-counter/README.txt
@@ -2,7 +2,7 @@ An application that displays the current top 10 topics, as gathered from the twi
It was ported and adapted from S4 0.3
Architecture:
-- twitter-adapter app in adapter node connects to the twitter stream, extracts the twitted text and passes that to the application cluster
+- twitter-adapter app in adapter node connects to the twitter stream, extracts the tweeted text and passes that to the application cluster
- twitter-counter app in the application cluster receives the text of the tweets, extracts the topics, counts topic occurences and periodically displays the top 10 topics on the console
How to configure:
@@ -12,22 +12,6 @@ user=<a twitter user name>
password=<the matching password>
How to run:
-0/ make sure tools are compiled by running ./gradlew s4-tools:installApp
-1/ start zookeeper server
-./s4 zkServer
-
-2/ create adapter cluster configuration
-./s4 newCluster -name=s4-test-cluster -firstListeningPort=10000 -nbTasks=1
-
-3/ create application cluster configuration
-./s4 newCluster -name=s4-adapter-cluster -firstListeningPort=11000 -nbTasks=<number of nodes>
-NOTE: - the name of the downstream cluster is currently hardcoded in <s4-dir>/test-apps/twitter-adapter/src/main/resources/s4.properties Make sure you use the same name
-
-4/ start application nodes (as many as defined in the cluster configuration, more for failover capabilities)
-./s4 appNode <s4-dir>/subprojects/s4-core/src/test/resources/default.s4.properties
-
-5/ start adapter node
-./s4 adapterNode -s4Properties=<s4-dir>/test-apps/twitter-adapter/src/main/resources/s4.properties
-
-6/ observe the topic count output (on 1 of the application nodes)
\ No newline at end of file
+Please follow the instructions in the S4 piper walkthrough at the following place:
+https://cwiki.apache.org/confluence/display/S4/S4+piper+walkthrough
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/94f54a19/test-apps/twitter-counter/src/main/resources/default.s4.properties
----------------------------------------------------------------------
diff --git a/test-apps/twitter-counter/src/main/resources/default.s4.properties b/test-apps/twitter-counter/src/main/resources/default.s4.properties
deleted file mode 100644
index 16fc4ba..0000000
--- a/test-apps/twitter-counter/src/main/resources/default.s4.properties
+++ /dev/null
@@ -1,19 +0,0 @@
-comm.queue_emmiter_size = 8000
-comm.queue_listener_size = 8000
-comm.retries=10
-comm.retry_delay=10
-comm.timeout=1000
-tcp.partition.queue_size=256
-cluster.hosts = localhost
-cluster.ports = 5077
-cluster.name = s4-adapter-cluster
-cluster.zk_address = localhost:21810
-cluster.zk_session_timeout = 10000
-cluster.zk_connection_timeout = 10000
-comm.module = org.apache.s4.core.CustomModule
-s4.logger_level = DEBUG
-appsDir=/tmp/deploy-test
-tcp.partition.queue_size=1000
-comm.timeout=100
-comm.retry_delay=100
-comm.retries=10