You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Juhani Connolly <ju...@gmail.com> on 2013/01/22 05:32:47 UTC
Review Request: Remove transaction capacity from Memory and File channels
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9052/
-----------------------------------------------------------
Review request for Flume.
Description
-------
Stops close() from removing the idle callback and skips close in the idle callback if the writer is already closed.
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3f31ef2
Diff: https://reviews.apache.org/r/9052/diff/
Testing
-------
Tests pass. Since sfWriters is protected it is hard to verify the changed behavior in unit tests. I however verified the behaviour in the java debugger.
Thanks,
Juhani Connolly
Re: Review Request: FLUME-1864: Allow hdfs idle callback to clean up closed
bucket writers
Posted by Juhani Connolly <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9052/#review16318
-----------------------------------------------------------
A review on this would be appreciated. We've reduced the default open files to 100 but this would also help with the underlying problem
- Juhani Connolly
On Jan. 24, 2013, 1:46 a.m., Juhani Connolly wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9052/
> -----------------------------------------------------------
>
> (Updated Jan. 24, 2013, 1:46 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Stops close() from removing the idle callback and skips close in the idle callback if the writer is already closed.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3f31ef2
>
> Diff: https://reviews.apache.org/r/9052/diff/
>
>
> Testing
> -------
>
> Tests pass. Since sfWriters is protected it is hard to verify the changed behavior in unit tests. I however verified the behaviour in the java debugger.
>
>
> Thanks,
>
> Juhani Connolly
>
>
Re: Review Request: FLUME-1864: Allow hdfs idle callback to clean up closed
bucket writers
Posted by Hari Shreedharan <hs...@cloudera.com>.
> On Feb. 8, 2013, 2:10 a.m., Juhani Connolly wrote:
> > A review on this would be appreciated. We've reduced the default open files to 100 but this would also help with the underlying problem
Done!
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9052/#review16319
-----------------------------------------------------------
On Jan. 24, 2013, 1:46 a.m., Juhani Connolly wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9052/
> -----------------------------------------------------------
>
> (Updated Jan. 24, 2013, 1:46 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Stops close() from removing the idle callback and skips close in the idle callback if the writer is already closed.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3f31ef2
>
> Diff: https://reviews.apache.org/r/9052/diff/
>
>
> Testing
> -------
>
> Tests pass. Since sfWriters is protected it is hard to verify the changed behavior in unit tests. I however verified the behaviour in the java debugger.
>
>
> Thanks,
>
> Juhani Connolly
>
>
Re: Review Request: FLUME-1864: Allow hdfs idle callback to clean up closed
bucket writers
Posted by Juhani Connolly <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9052/#review16319
-----------------------------------------------------------
A review on this would be appreciated. We've reduced the default open files to 100 but this would also help with the underlying problem
- Juhani Connolly
On Jan. 24, 2013, 1:46 a.m., Juhani Connolly wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9052/
> -----------------------------------------------------------
>
> (Updated Jan. 24, 2013, 1:46 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Stops close() from removing the idle callback and skips close in the idle callback if the writer is already closed.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3f31ef2
>
> Diff: https://reviews.apache.org/r/9052/diff/
>
>
> Testing
> -------
>
> Tests pass. Since sfWriters is protected it is hard to verify the changed behavior in unit tests. I however verified the behaviour in the java debugger.
>
>
> Thanks,
>
> Juhani Connolly
>
>
Re: Review Request: FLUME-1864: Allow hdfs idle callback to clean up closed
bucket writers
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9052/#review16670
-----------------------------------------------------------
Ship it!
Ship It!
- Hari Shreedharan
On Jan. 24, 2013, 1:46 a.m., Juhani Connolly wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9052/
> -----------------------------------------------------------
>
> (Updated Jan. 24, 2013, 1:46 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Stops close() from removing the idle callback and skips close in the idle callback if the writer is already closed.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3f31ef2
>
> Diff: https://reviews.apache.org/r/9052/diff/
>
>
> Testing
> -------
>
> Tests pass. Since sfWriters is protected it is hard to verify the changed behavior in unit tests. I however verified the behaviour in the java debugger.
>
>
> Thanks,
>
> Juhani Connolly
>
>
Re: Review Request: FLUME-1864: Allow hdfs idle callback to clean up closed
bucket writers
Posted by Juhani Connolly <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9052/
-----------------------------------------------------------
(Updated Jan. 24, 2013, 1:46 a.m.)
Review request for Flume.
Summary (updated)
-----------------
FLUME-1864: Allow hdfs idle callback to clean up closed bucket writers
Description
-------
Stops close() from removing the idle callback and skips close in the idle callback if the writer is already closed.
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3f31ef2
Diff: https://reviews.apache.org/r/9052/diff/
Testing
-------
Tests pass. Since sfWriters is protected it is hard to verify the changed behavior in unit tests. I however verified the behaviour in the java debugger.
Thanks,
Juhani Connolly
Re: Review Request: Allow hdfs idle callback to clean up closed bucket
writers
Posted by Juhani Connolly <ju...@gmail.com>.
> On Jan. 22, 2013, 4:55 a.m., Roshan Naik wrote:
> > Absence of both transactionCapacity and batchsize information would make things difficult for the spillable transaction [flume 1227].
> >
> > if batch size is exposed to the transaction i am ok with dropping this transaction capacity in file and memory channels.
I misnamed this initially by accident. This review is just for a change in the behavior of idle callback. At this time I don't plan to work on 1860
- Juhani
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9052/#review15560
-----------------------------------------------------------
On Jan. 22, 2013, 4:34 a.m., Juhani Connolly wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9052/
> -----------------------------------------------------------
>
> (Updated Jan. 22, 2013, 4:34 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Stops close() from removing the idle callback and skips close in the idle callback if the writer is already closed.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3f31ef2
>
> Diff: https://reviews.apache.org/r/9052/diff/
>
>
> Testing
> -------
>
> Tests pass. Since sfWriters is protected it is hard to verify the changed behavior in unit tests. I however verified the behaviour in the java debugger.
>
>
> Thanks,
>
> Juhani Connolly
>
>
Re: Review Request: Allow hdfs idle callback to clean up closed bucket
writers
Posted by Roshan Naik <ro...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9052/#review15560
-----------------------------------------------------------
Absence of both transactionCapacity and batchsize information would make things difficult for the spillable transaction [flume 1227].
if batch size is exposed to the transaction i am ok with dropping this transaction capacity in file and memory channels.
- Roshan Naik
On Jan. 22, 2013, 4:34 a.m., Juhani Connolly wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9052/
> -----------------------------------------------------------
>
> (Updated Jan. 22, 2013, 4:34 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Stops close() from removing the idle callback and skips close in the idle callback if the writer is already closed.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3f31ef2
>
> Diff: https://reviews.apache.org/r/9052/diff/
>
>
> Testing
> -------
>
> Tests pass. Since sfWriters is protected it is hard to verify the changed behavior in unit tests. I however verified the behaviour in the java debugger.
>
>
> Thanks,
>
> Juhani Connolly
>
>
Re: Review Request: Allow hdfs idle callback to clean up closed bucket
writers
Posted by Juhani Connolly <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9052/
-----------------------------------------------------------
(Updated Jan. 22, 2013, 4:34 a.m.)
Review request for Flume.
Summary (updated)
-----------------
Allow hdfs idle callback to clean up closed bucket writers
Description
-------
Stops close() from removing the idle callback and skips close in the idle callback if the writer is already closed.
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3f31ef2
Diff: https://reviews.apache.org/r/9052/diff/
Testing
-------
Tests pass. Since sfWriters is protected it is hard to verify the changed behavior in unit tests. I however verified the behaviour in the java debugger.
Thanks,
Juhani Connolly