You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Edi Bice <ed...@yahoo.com> on 2016/02/25 20:39:49 UTC

Re: Review Request 43732: Implemented AvroDataFileHdfsWriter

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43732/
-----------------------------------------------------------

(Updated Feb. 25, 2016, 7:39 p.m.)


Review request for samza.


Changes
-------

Reducing the scope as Yi recommended (moving the RocksDb changes to a separate JIRA/patch)


Summary (updated)
-----------------

Implemented AvroDataFileHdfsWriter


Repository: samza


Description (updated)
-------

https://issues.apache.org/jira/browse/SAMZA-876 

Implemented AvroDataFileHdfsWriter fashioned loosely after BinarySequenceFileHDFSWriter.


Diffs
-----

  docs/learn/documentation/versioned/hdfs/producer.md cfd22c6 
  docs/learn/documentation/versioned/jobs/configuration-table.html 6705530 
  gradle/dependency-versions.gradle 52e25aa 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala 7993119 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/AvroDataFileHdfsWriter.scala PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-avro.properties PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-job-avro.properties PRE-CREATION 
  samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala c4b04a1 

Diff: https://reviews.apache.org/r/43732/diff/


Testing (updated)
-------

Two JUnit tests similar to the Text/BinarySequenceFileHdfsWriter ones. In addition I've been using AvroDataFileHdfsWriter at the end of my pipeline. I feed the generated avro files to Apache Samoa. Have processed millions of records successfully.


Thanks,

Edi Bice

Re: Review Request 43732: Implemented AvroDataFileHdfsWriter

Posted by Edi Bice <ed...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43732/
-----------------------------------------------------------

(Updated March 3, 2016, 6:58 p.m.)


Review request for samza.


Changes
-------

Another fresh rebase since Jackson has been upgraded in master


Repository: samza


Description
-------

https://issues.apache.org/jira/browse/SAMZA-876 

Implemented AvroDataFileHdfsWriter fashioned loosely after BinarySequenceFileHDFSWriter.


Diffs (updated)
-----

  docs/learn/documentation/versioned/hdfs/producer.md cfd22c6 
  docs/learn/documentation/versioned/jobs/configuration-table.html 175437c 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala 7993119 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/AvroDataFileHdfsWriter.scala PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-avro.properties PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-job-avro.properties PRE-CREATION 
  samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala c4b04a1 

Diff: https://reviews.apache.org/r/43732/diff/


Testing
-------

Two JUnit tests similar to the Text/BinarySequenceFileHdfsWriter ones. In addition I've been using AvroDataFileHdfsWriter at the end of my pipeline. I feed the generated avro files to Apache Samoa. Have processed millions of records successfully.


Thanks,

Edi Bice

Re: Review Request 43732: Implemented AvroDataFileHdfsWriter

Posted by Edi Bice <ed...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43732/
-----------------------------------------------------------

(Updated March 2, 2016, 4:35 p.m.)


Review request for samza.


Changes
-------

Rebased with latest from master branch as requested


Repository: samza


Description
-------

https://issues.apache.org/jira/browse/SAMZA-876 

Implemented AvroDataFileHdfsWriter fashioned loosely after BinarySequenceFileHDFSWriter.


Diffs (updated)
-----

  docs/learn/documentation/versioned/hdfs/producer.md cfd22c6 
  docs/learn/documentation/versioned/jobs/configuration-table.html 6705530 
  gradle/dependency-versions.gradle 52e25aa 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala 7993119 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/AvroDataFileHdfsWriter.scala PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-avro.properties PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-job-avro.properties PRE-CREATION 
  samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala c4b04a1 

Diff: https://reviews.apache.org/r/43732/diff/


Testing
-------

Two JUnit tests similar to the Text/BinarySequenceFileHdfsWriter ones. In addition I've been using AvroDataFileHdfsWriter at the end of my pipeline. I feed the generated avro files to Apache Samoa. Have processed millions of records successfully.


Thanks,

Edi Bice

Re: Review Request 43732: Implemented AvroDataFileHdfsWriter

Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43732/#review121594
-----------------------------------------------------------



Overall lgtm. There is some problem in the last file in the diff uploaded. Could you try to rebase and upload again?Thanks!


gradle/dependency-versions.gradle (line 23)
<https://reviews.apache.org/r/43732/#comment183340>

    Please rebase w/ the latest master branch.


- Yi Pan (Data Infrastructure)


On Feb. 25, 2016, 7:39 p.m., Edi Bice wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43732/
> -----------------------------------------------------------
> 
> (Updated Feb. 25, 2016, 7:39 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> https://issues.apache.org/jira/browse/SAMZA-876 
> 
> Implemented AvroDataFileHdfsWriter fashioned loosely after BinarySequenceFileHDFSWriter.
> 
> 
> Diffs
> -----
> 
>   docs/learn/documentation/versioned/hdfs/producer.md cfd22c6 
>   docs/learn/documentation/versioned/jobs/configuration-table.html 6705530 
>   gradle/dependency-versions.gradle 52e25aa 
>   samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala 7993119 
>   samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/AvroDataFileHdfsWriter.scala PRE-CREATION 
>   samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-avro.properties PRE-CREATION 
>   samza-hdfs/src/test/resources/samza-hdfs-test-job-avro.properties PRE-CREATION 
>   samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala c4b04a1 
> 
> Diff: https://reviews.apache.org/r/43732/diff/
> 
> 
> Testing
> -------
> 
> Two JUnit tests similar to the Text/BinarySequenceFileHdfsWriter ones. In addition I've been using AvroDataFileHdfsWriter at the end of my pipeline. I feed the generated avro files to Apache Samoa. Have processed millions of records successfully.
> 
> 
> Thanks,
> 
> Edi Bice
> 
>