You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by xi...@apache.org on 2017/02/02 22:33:17 UTC
samza git commit: Fix javadoc issues introduced by Hdfs consumer
Repository: samza
Updated Branches:
refs/heads/master fcb44be80 -> 1f4a59328
Fix javadoc issues introduced by Hdfs consumer
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/1f4a5932
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/1f4a5932
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/1f4a5932
Branch: refs/heads/master
Commit: 1f4a593288ef5e216377777adb30d91e442c4d7c
Parents: fcb44be
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Authored: Thu Feb 2 14:31:23 2017 -0800
Committer: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Committed: Thu Feb 2 14:33:00 2017 -0800
----------------------------------------------------------------------
docs/learn/documentation/versioned/hdfs/consumer.md | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/1f4a5932/docs/learn/documentation/versioned/hdfs/consumer.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/hdfs/consumer.md b/docs/learn/documentation/versioned/hdfs/consumer.md
index 401b228..41a1a51 100644
--- a/docs/learn/documentation/versioned/hdfs/consumer.md
+++ b/docs/learn/documentation/versioned/hdfs/consumer.md
@@ -19,7 +19,7 @@ title: Reading from HDFS
limitations under the License.
-->
-You can configure your Samza job to read from HDFS files. The [HdfsSystemConsumer](javadocs/org/apache/samza/system/hdfs/HdfsSystemConsumer.html) can read from HDFS files. Avro encoded records are supported out of the box and it is easy to extend to support other formats (plain text, csv, json etc). See `Event format` section below.
+You can configure your Samza job to read from HDFS files. The [HdfsSystemConsumer](https://github.com/apache/samza/blob/master/samza-hdfs/src/main/java/org/apache/samza/system/hdfs/HdfsSystemConsumer.java) can read from HDFS files. Avro encoded records are supported out of the box and it is easy to extend to support other formats (plain text, csv, json etc). See `Event format` section below.
### Environment
@@ -31,18 +31,19 @@ Partitioning works at the level of individual HDFS files. Each file is treated a
### Event format
-[HdfsSystemConsumer](javadocs/org/apache/samza/system/hdfs/HdfsSystemConsumer.html) currently supports reading from avro files. The received [IncomingMessageEnvelope](javadocs/org/apache/samza/system/IncomingMessageEnvelope.html) contains three significant fields:
+[HdfsSystemConsumer](https://github.com/apache/samza/blob/master/samza-hdfs/src/main/java/org/apache/samza/system/hdfs/HdfsSystemConsumer.java) currently supports reading from avro files. The received [IncomingMessageEnvelope](../api/javadocs/org/apache/samza/system/IncomingMessageEnvelope.html) contains three significant fields:
+
1. The key which is empty
2. The message which is set to the avro [GenericRecord](https://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/generic/GenericRecord.html)
3. The stream partition which is set to the name of the HDFS file
-To extend the support beyond avro files (e.g. json, csv, etc.), you can implement the interface [SingleFileHdfsReader](javadocs/org/apache/samza/system/hdfs/reader/SingleFileHdfsReader.html) (take a look at the implementation of [AvroFileHdfsReader](javadocs/org/apache/samza/system/hdfs/reader/AvroFileHdfsReader) as a sample).
+To extend the support beyond avro files (e.g. json, csv, etc.), you can implement the interface [SingleFileHdfsReader](https://github.com/apache/samza/blob/master/samza-hdfs/src/main/java/org/apache/samza/system/hdfs/reader/SingleFileHdfsReader.java) (take a look at the implementation of [AvroFileHdfsReader](https://github.com/apache/samza/blob/master/samza-hdfs/src/main/java/org/apache/samza/system/hdfs/reader/AvroFileHdfsReader.java) as a sample).
### End of stream support
One�major�difference�between HDFS�data and Kafka data is�that while a kafka topic has an unbounded stream of messages, HDFS files are bounded and have a notion of EOF.
-You can choose to implement [EndOfStreamListenerTask](javadocs/org/apache/samza/task/EndOfStreamListenerTask.html) to receive a callback when all partitions are at end of stream. When all partitions being processed by the task are at end of stream (i.e. EOF has been reached for all files), the Samza job exits automatically.
+You can choose to implement [EndOfStreamListenerTask](../api/javadocs/org/apache/samza/task/EndOfStreamListenerTask.html) to receive a callback when all partitions are at end of stream. When all partitions being processed by the task are at end of stream (i.e. EOF has been reached for all files), the Samza job exits automatically.
### Basic Configuration