You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Eli Reisman <in...@gmail.com> on 2015/07/18 19:31:23 UTC
Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service
for Samza
> On June 25, 2015, 6:35 p.m., Yan Fang wrote:
> >
Updating this patch today, should be pushing a new patch sometime this weekend if all goes well, sorry for the delay
- Eli
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35445/#review89390
-----------------------------------------------------------
On June 14, 2015, 10:17 p.m., Eli Reisman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35445/
> -----------------------------------------------------------
>
> (Updated June 14, 2015, 10:17 p.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA-693: Very basic HDFS Producer service for Samza
>
>
> Diffs
> -----
>
> build.gradle a5f54106a822dc91ff82270df27217a8765a0d80
> samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala PRE-CREATION
> samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemAdmin.scala PRE-CREATION
> samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemFactory.scala PRE-CREATION
> samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducer.scala PRE-CREATION
> samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducerMetrics.scala PRE-CREATION
> samza-hdfs/src/test/org/apache/samza/system/hdfs/TestHdfsSystemProducer.scala PRE-CREATION
> samza-hdfs/src/test/resources/samza-hdfs-test-job.properties PRE-CREATION
> settings.gradle bb07a3b84b14dcef94da1bb166eab6aa3d0026bb
>
> Diff: https://reviews.apache.org/r/35445/diff/
>
>
> Testing
> -------
>
> New unit test, but it's fairly rudimentary. Passes "./gradlew test" and "./gradlew check"
>
> This only supplies an HDFS Producer, and this producer only writes SequenceFiles of ByteWriteables so far. If the patch were accepted as-is, I'd suggest future tickets for a matching HDFS Consumer, and a pluggable set of output formats, configurable via HdfsConfig settings.
>
> On the upside, this patch has been tested on a real cluster with real data, using several serdes, with good results.
>
>
> Thanks,
>
> Eli Reisman
>
>