You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Ted Malaska <te...@cloudera.com> on 2016/11/01 14:48:05 UTC
Review Request 53339: FLUME-3020: Improve HDFSEventSink Escape
Ingestion by more then 10x by not getting InetAddress on every record
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53339/
-----------------------------------------------------------
Review request for Flume and Jeff Holoman.
Bugs: FLUME-3020
https://issues.apache.org/jira/browse/FLUME-3020
Repository: flume-git
Description
-------
If you are use escaping the current code will call InetAddress on every record. Which will result is a huge impact to performance.
TotalTime,8403,
totalEventTakeTime,1498,
totalWriteTime,1981,
totalWriterSetupTime,65,
commitTime,201,
flushTime,18,
startTrans,7,
The rest is all InetAddress
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java cf0fbb0
Diff: https://reviews.apache.org/r/53339/diff/
Testing
-------
Thanks,
Ted Malaska
Re: Review Request 53339: FLUME-3020: Improve HDFSEventSink Escape
Ingestion by more then 10x by not getting InetAddress on every record
Posted by Balázs Donát Bessenyei <be...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53339/#review155519
-----------------------------------------------------------
Ship it!
Ship It!
- Bal�zs Don�t Bessenyei
On nov. 9, 2016, 10:03 du, Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53339/
> -----------------------------------------------------------
>
> (Updated nov. 9, 2016, 10:03 du)
>
>
> Review request for Flume and Jeff Holoman.
>
>
> Bugs: FLUME-3020
> https://issues.apache.org/jira/browse/FLUME-3020
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> If you are use escaping the current code will call InetAddress on every record. Which will result is a huge impact to performance.
> TotalTime,8403,
> totalEventTakeTime,1498,
> totalWriteTime,1981,
> totalWriterSetupTime,65,
> commitTime,201,
> flushTime,18,
> startTrans,7,
> The rest is all InetAddress
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java cf0fbb0
>
> Diff: https://reviews.apache.org/r/53339/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>
Re: Review Request 53339: FLUME-3020: Improve HDFSEventSink Escape
Ingestion by more then 10x by not getting InetAddress on every record
Posted by Ted Malaska <te...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53339/
-----------------------------------------------------------
(Updated Nov. 9, 2016, 10:03 p.m.)
Review request for Flume and Jeff Holoman.
Changes
-------
Added private and final to static class InetAddressCache
Bugs: FLUME-3020
https://issues.apache.org/jira/browse/FLUME-3020
Repository: flume-git
Description
-------
If you are use escaping the current code will call InetAddress on every record. Which will result is a huge impact to performance.
TotalTime,8403,
totalEventTakeTime,1498,
totalWriteTime,1981,
totalWriterSetupTime,65,
commitTime,201,
flushTime,18,
startTrans,7,
The rest is all InetAddress
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java cf0fbb0
Diff: https://reviews.apache.org/r/53339/diff/
Testing
-------
Thanks,
Ted Malaska
Re: Review Request 53339: FLUME-3020: Improve HDFSEventSink Escape
Ingestion by more then 10x by not getting InetAddress on every record
Posted by Ted Malaska <te...@cloudera.com>.
> On Nov. 3, 2016, 1:58 p.m., Jeff Holoman wrote:
> > flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java, line 552
> > <https://reviews.apache.org/r/53339/diff/2/?file=1550881#file1550881line552>
> >
> > nit: can you make this final
>
> Bal�zs Don�t Bessenyei wrote:
> Maybe (even) private?
Will update today
- Ted
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53339/#review154723
-----------------------------------------------------------
On Nov. 1, 2016, 8:16 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53339/
> -----------------------------------------------------------
>
> (Updated Nov. 1, 2016, 8:16 p.m.)
>
>
> Review request for Flume and Jeff Holoman.
>
>
> Bugs: FLUME-3020
> https://issues.apache.org/jira/browse/FLUME-3020
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> If you are use escaping the current code will call InetAddress on every record. Which will result is a huge impact to performance.
> TotalTime,8403,
> totalEventTakeTime,1498,
> totalWriteTime,1981,
> totalWriterSetupTime,65,
> commitTime,201,
> flushTime,18,
> startTrans,7,
> The rest is all InetAddress
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java cf0fbb0
>
> Diff: https://reviews.apache.org/r/53339/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>
Re: Review Request 53339: FLUME-3020: Improve HDFSEventSink Escape
Ingestion by more then 10x by not getting InetAddress on every record
Posted by Balázs Donát Bessenyei <be...@apache.org>.
> On nov. 3, 2016, 1:58 du, Jeff Holoman wrote:
> > flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java, line 552
> > <https://reviews.apache.org/r/53339/diff/2/?file=1550881#file1550881line552>
> >
> > nit: can you make this final
Maybe (even) private?
- Bal�zs Don�t
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53339/#review154723
-----------------------------------------------------------
On nov. 1, 2016, 8:16 du, Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53339/
> -----------------------------------------------------------
>
> (Updated nov. 1, 2016, 8:16 du)
>
>
> Review request for Flume and Jeff Holoman.
>
>
> Bugs: FLUME-3020
> https://issues.apache.org/jira/browse/FLUME-3020
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> If you are use escaping the current code will call InetAddress on every record. Which will result is a huge impact to performance.
> TotalTime,8403,
> totalEventTakeTime,1498,
> totalWriteTime,1981,
> totalWriterSetupTime,65,
> commitTime,201,
> flushTime,18,
> startTrans,7,
> The rest is all InetAddress
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java cf0fbb0
>
> Diff: https://reviews.apache.org/r/53339/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>
Re: Review Request 53339: FLUME-3020: Improve HDFSEventSink Escape
Ingestion by more then 10x by not getting InetAddress on every record
Posted by Jeff Holoman <je...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53339/#review154723
-----------------------------------------------------------
flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java (line 547)
<https://reviews.apache.org/r/53339/#comment224391>
nit: can you make this final
- Jeff Holoman
On Nov. 1, 2016, 8:16 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53339/
> -----------------------------------------------------------
>
> (Updated Nov. 1, 2016, 8:16 p.m.)
>
>
> Review request for Flume and Jeff Holoman.
>
>
> Bugs: FLUME-3020
> https://issues.apache.org/jira/browse/FLUME-3020
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> If you are use escaping the current code will call InetAddress on every record. Which will result is a huge impact to performance.
> TotalTime,8403,
> totalEventTakeTime,1498,
> totalWriteTime,1981,
> totalWriterSetupTime,65,
> commitTime,201,
> flushTime,18,
> startTrans,7,
> The rest is all InetAddress
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java cf0fbb0
>
> Diff: https://reviews.apache.org/r/53339/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>
Re: Review Request 53339: FLUME-3020: Improve HDFSEventSink Escape
Ingestion by more then 10x by not getting InetAddress on every record
Posted by Ted Malaska <te...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53339/
-----------------------------------------------------------
(Updated Nov. 1, 2016, 8:16 p.m.)
Review request for Flume and Jeff Holoman.
Changes
-------
Added two things. Now it caches the string for reuse and it also lazy loads the fetching of the information
Bugs: FLUME-3020
https://issues.apache.org/jira/browse/FLUME-3020
Repository: flume-git
Description
-------
If you are use escaping the current code will call InetAddress on every record. Which will result is a huge impact to performance.
TotalTime,8403,
totalEventTakeTime,1498,
totalWriteTime,1981,
totalWriterSetupTime,65,
commitTime,201,
flushTime,18,
startTrans,7,
The rest is all InetAddress
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java cf0fbb0
Diff: https://reviews.apache.org/r/53339/diff/
Testing
-------
Thanks,
Ted Malaska