You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flume.apache.org by pr...@apache.org on 2011/12/16 02:10:41 UTC
svn commit: r1215014 - in /incubator/flume/branches/flume-728: DEVNOTES
NOTICE README
Author: prasadm
Date: Fri Dec 16 01:10:40 2011
New Revision: 1215014
URL: http://svn.apache.org/viewvc?rev=1215014&view=rev
Log:
FLUME-876. Update README, DEVNOTES and NOTICE
Modified:
incubator/flume/branches/flume-728/DEVNOTES
incubator/flume/branches/flume-728/NOTICE
incubator/flume/branches/flume-728/README
Modified: incubator/flume/branches/flume-728/DEVNOTES
URL: http://svn.apache.org/viewvc/incubator/flume/branches/flume-728/DEVNOTES?rev=1215014&r1=1215013&r2=1215014&view=diff
==============================================================================
--- incubator/flume/branches/flume-728/DEVNOTES (original)
+++ incubator/flume/branches/flume-728/DEVNOTES Fri Dec 16 01:10:40 2011
@@ -1,7 +1,5 @@
Flume Developer Notes
=====================
-Jonathan Hsieh <jm...@apache.org>
-6/22/11
// This is in asciidoc markup
@@ -19,45 +17,21 @@ we describe the contents of different di
----
./bin/ Flume startup scripts
./conf/ Flume configuration file samples
-./contrib/flogger Flume logger: a Flume client implemented in C
-./docs/man Flume man pages
-./flume-config-web Flume master configuration servlet module
-./flume-core Flume core module
-./flume-distribution Flume distribution package module
-./flume-docs Flume documentation generation module
-./flume-log4j-appender Flume log4j-avro appender module
-./flume-microbenchmarks Flume performance microbenchmark test suite
-./flume-node-web Flume node status servlet module
-./flume-windows-dist Flume node Windows distribution package module
-./plugins/ Flume plugin modules (hello world skeleton and hbase)
-./src/javaperf Flume performance tests (out of date)
-./src/javatest-torture Flume reliability tests (out of date)
+./flume-ng-core Flume core module
+./flume-ng-dist Flume distribution package module
+./flume-ng-channels Flume channels modules (memory, JDBC and File)
+./flume-ng-node Flume node module
+./flume-ng-sinks Flume sink modules (HDFS)
----
The files exclusions in `.gitignore` are either autogenerated by Maven or Eclipse.
== Building and Testing Flume
-=== Prerequisites
-
-There are several tools required to do a full build of Flume but only
-the Thrift compiler is required for development and testing builds.
-
-To build documentation, you will need to have asciidoc installed.
-
-To build Windows installers, you will need to have makensis installed.
-==== Building Thrift
-
-The Thrift compiler is required to build Flume and currently does not
-have a binary packages avaiblle for Linux based platforms. (Windows
-is available in binary). There are several requirements necesary to
-build it. Here's a link to the requirements
-
-http://wiki.apache.org/thrift/ThriftRequirements
+=== Prerequisites
+You need to have Apache Maven installed.
-This page also contains links explaining how to install the
-requirements for various platforms.
=== Using Maven
@@ -71,9 +45,7 @@ The default profile is a "dev" profile.
of common build command lines to build different profiles.
A development build that runs unit tests and installs to local Maven
-repo. This builds and tests all plugins, but excludes modules that
-have aren't needed during development (eg. Windows installer,
-documentation).
+repo. This builds and tests all plugins.
----
mvn install
@@ -99,70 +71,6 @@ or path).
mvn test -Dtest=<ClassRegex>
----
-Window node build, skipping unit tests (requires makensis).
-
-NOTE: makensis is available on Linux and Mac OS X homebrew so this can
-be built while running in these operating systems.
-
-----
-mvn install -Pwindows -DskipTests
-----
-
-Full build, skipping unit tests (requires asciidoc), and does not build Windows.
-
-----
-mvn install -Pfull-build -DskipTests
-----
-
-Full build, make both docs and Windows.
-
-----
-mvn install -Pfull-build,windows
-----
-
-==== Pointing the Maven build at the proper Thrift executable
-
-Flume has, over time, upgraded to newer versions of Thrift. The Maven
-build requires a pointer to the proper Thrift compiler.
-
-If you install Thrift in a non-standard location (not
-/usr/local/thrift/bin), you will need to provide the build some extra
-information. This may be the case if you overrode the standard Thrift
-install (+make install+ 's default target) or are running Thrift from
-a home directory.
-
-One way to provide this is via the Maven command line by setting the
-thrift.executable variable (this assumes that we made different dirs
-for different versions of Thrift):
-
-----
-mvn install -Dthrift.executable=/usr/local/thrift-0.6.0/bin/thrift
-----
-
-Another way to provide this information to your Maven build is to
-modifiy your Maven profile by adding/modifiying your
-~/.m2/settings.xml file and overriding the default thrift.executable
-setting to point to your Thrift compiler executable. In the example
-below, we install different versions of the Thrift compiler in
-different directories and thus need to change the setting.
-
-----
-<settings>
- <profiles>
- <profile>
- <id>flume</id>
- <properties>
- <thrift.executable>/usr/local/thrift-0.6.0/bin/thrift</thrift.executable>
- </properties>
- </profile>
- </profiles>
-
- <activeProfiles>
- <activeProfile>flume</activeProfile>
- </activeProfiles>
-</settings>
-----
-
==== Including or excluding specific sets of tests.
We've added hooks to the maven build that will enable you to exclude
@@ -225,62 +133,6 @@ packages.
mvn install -DskipTests
----
-You can then traverse into
-./flume-distribution/target/flume-distribution-<version>-bin/flume-distribution-<version>.
-This directory is setup exactly as the tarball installation of Flume
-would be.
-
-
-=== Running Performance Microbenchmarks.
-
-The suite of source and sink microbenchmark tests (located in
-./flume-microbenchmarks/javaperf) can be run by using `mvn test -Pperf`.
-
-Just like with the normal test cases, you can use the
-`-Dtest=<TestClass>`. So you can do:
-
-----
-mvn test -Pperf -Dtest=PerfThriftSinks
-----
-
-The logs should output lines that are formatted similarly to these
-lines:
-
-----
-[junit] nullsink,ubuntu,begin,10998597,552872,disk_loaded,2895851957,301662152,receiver_started,156786445,305698624,sink_started,105303802,305704456,thrift sink to thrift source done,39520160510,320377056,MB/s,4.579940971898899,23094932,320379168
- [junit] [ 0us, 547,544 b mem] Starting (after gc)
- [junit] [ 10,998,597ns d 10,998,597ns 552,872 b mem] begin
- [junit] [ 2,914,443,637ns d 2,895,851,957ns 301,662,152 b mem] disk_loaded
- [junit] [ 3,514,297,391ns d 156,786,445ns 305,698,624 b mem] receiver_started
- [junit] [ 4,082,661,503ns d 105,303,802ns 305,704,456 b mem] sink_started
- [junit] [ 44,235,264,972ns d 39,520,160,510ns 320,377,056 b mem] thrift sink to thrift source done
- [junit] [ 44,878,445,315ns d 23,094,932ns 320,379,168 b mem] MB/s,4.579940971898899
-----
-
-The first line is a summary of all the information in cvs format. The
-other lines are in a tabular, more human-readable form. The left
-column is cumulative time in ns and the middle is delta from previous
-in ns. The last column of numbers the amount of memory in heap,
-followed but some comments or labels.
-
-
-=== Building on Windows platforms
-
-Building Flume in Windows is possible. One can generate packages and
-installer executable on Windows. This build assumes a cygwin
-envrionment, but may not require it.
-
-This build requires
-
-* Maven for Windows
-* makensis (for Windows installer build)
-* java 1.6+
-
-You should be able run the normal mvn commands.
-
-The current Windows installer executable does not handle all error
-handling situations and does not checks to see if not run as
-administrator.
=== Building documentation
@@ -313,17 +165,6 @@ This can be done by going to the Eclipse
to File > Import... > Existing Maven Projects. From there, browse to
and select the directory that contains the root of the Flume project.
-The build requires the location of the Thrift compiler executable --
-see the instructions about .m2/settings.xml files in the building
-Flume section for more details.
-
-The flume-core project will have errors -- these can be resolved by manually adding these dirs to you build source dirs:
-
-* ./flume-core/target/generated-sources/antlr3
-* ./flume-core/target/generated-sources/avro
-* ./flume-core/target/generated-sources/thrift
-* ./flume-core/target/generated-sources/version
-
== Debugging Flume
=== Flume's web applications
@@ -379,21 +220,22 @@ depending on comment characters):
----
/**
- * Licensed to Cloudera, Inc. under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. Cloudera, Inc. licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
*/
----
@@ -401,8 +243,5 @@ No build generated files should be check
of generate files that should not be checked:
* html documentation
-* thrift-generated source
* avro-generated source
-* antlr generated source
* auto-generated versioning annotations
-
Modified: incubator/flume/branches/flume-728/NOTICE
URL: http://svn.apache.org/viewvc/incubator/flume/branches/flume-728/NOTICE?rev=1215014&r1=1215013&r2=1215014&view=diff
==============================================================================
--- incubator/flume/branches/flume-728/NOTICE (original)
+++ incubator/flume/branches/flume-728/NOTICE Fri Dec 16 01:10:40 2011
@@ -1,3 +1,8 @@
-This product includes software developed by Cloudera, Inc.
-(http://www.cloudera.com/).
+Apache Flume
+Copyright 2011 The Apache Software Foundation
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+Portions of this software were developed at
+Cloudera, Inc. (http://www.cloudera.com/).
Modified: incubator/flume/branches/flume-728/README
URL: http://svn.apache.org/viewvc/incubator/flume/branches/flume-728/README?rev=1215014&r1=1215013&r2=1215014&view=diff
==============================================================================
--- incubator/flume/branches/flume-728/README (original)
+++ incubator/flume/branches/flume-728/README Fri Dec 16 01:10:40 2011
@@ -7,32 +7,27 @@ data flows. It is robust and fault toler
mechanisms and many failover and recovery mechanisms. The system is
centrally managed and allows for intelligent dynamic management. It
uses a simple extensible data model that allows for online analytic
-application
+application.
+The Flume-NG is refactoring of the first generation Flume to solve
+certain known issues and limitations of the original design.
Flume is open-sourced under the Apache Software Foundation License v2.0.
h2. Documentation
-Flume's installation instructions can be found here:
+Flume NG guide and FAQ are available here:
-https://docs.cloudera.com/display/DOC/Flume+Installation
-
-Flume's user guide, cookbook, and FAQ are available here:
-
-* http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html
-* http://archive.cloudera.com/cdh/3/flume/Cookbook/index.html
-* http://github.com/cloudera/flume/wiki/FAQ
+* https://cwiki.apache.org/FLUME/flume-ng.html
+* https://cwiki.apache.org/confluence/display/FLUME/Getting+Started
h2. Contact us!
-* User mailing list: https://groups.google.com/a/cloudera.org/group/flume-user/topics
-* Developer mailling list: https://groups.google.com/a/cloudera.org/group/flume-dev/topics
+* Mailing lists: http://incubator.apache.org/flume/mail-lists.html
* IRC channel #flume on irc.freenode.net
Bug and Issue tracker.
-* http://issues.cloudera.org/
-* https://issues.cloudera.org/browse/FLUME
+* https://issues.apache.org/jira/browse/FLUME
h2. Compiling Flume
@@ -44,4 +39,4 @@ Compiling Flume requires the following t
To compile Flume, run `mvn compile`.
To build a distribution, run `mvn package`.
-The final Flume distribution artifact will be in $project/flume-distribution/target/.
+The final Flume distribution artifact will be in $project/flume-ng-dist/target/.