You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Benjamin Smith <be...@ranksoftwareinc.com> on 2016/06/16 18:59:05 UTC
Review Request 48811: SAMZA-968 - SequenceFileHdfsFileWriter does not
close file properly
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48811/
-----------------------------------------------------------
Review request for samza.
Repository: samza
Description
-------
Create writer with appropriate API to ensure stream is closed with writer, and not just flushed
Diffs
-----
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/SequenceFileHdfsWriter.scala 46ade333af6acf1f2646924854bc2290ff56b5cb
Diff: https://reviews.apache.org/r/48811/diff/
Testing
-------
Unit tests pass successfully.
Performing manual test in our code base to ensure that files close properly (we currently see them remain open and several hour delays in propogating writes to HDFS)
Open to suggestions for how to write a unit test for this use case.
Thanks,
Benjamin Smith
Re: Review Request 48811: SAMZA-968 - SequenceFileHdfsFileWriter does
not close file properly
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48811/#review138622
-----------------------------------------------------------
Ship it!
lgtm. As for the unit test, is it possible to access the file handle within the writer object and after calling writer.close(), make sure that the file is closed as well?
- Yi Pan (Data Infrastructure)
On June 16, 2016, 6:59 p.m., Benjamin Smith wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48811/
> -----------------------------------------------------------
>
> (Updated June 16, 2016, 6:59 p.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> Create writer with appropriate API to ensure stream is closed with writer, and not just flushed
>
>
> Diffs
> -----
>
> samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/SequenceFileHdfsWriter.scala 46ade333af6acf1f2646924854bc2290ff56b5cb
>
> Diff: https://reviews.apache.org/r/48811/diff/
>
>
> Testing
> -------
>
> Unit tests pass successfully.
> Performing manual test in our code base to ensure that files close properly (we currently see them remain open and several hour delays in propogating writes to HDFS)
>
> Open to suggestions for how to write a unit test for this use case.
>
>
> Thanks,
>
> Benjamin Smith
>
>