You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/10/29 21:35:09 UTC
[GitHub] [hudi] bvaradar opened a new pull request #2217: [HUDI-1358] Fix Memory Leak in HoodieLogFormatWriter
bvaradar opened a new pull request #2217:
URL: https://github.com/apache/hudi/pull/2217
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar merged pull request #2217: [HUDI-1358] Fix Memory Leak in HoodieLogFormatWriter
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #2217:
URL: https://github.com/apache/hudi/pull/2217
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #2217: [HUDI-1358] Fix Memory Leak in HoodieLogFormatWriter
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2217:
URL: https://github.com/apache/hudi/pull/2217#discussion_r514640834
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java
##########
@@ -55,6 +55,8 @@
private final String rolloverLogWriteToken;
private FSDataOutputStream output;
private boolean closed = false;
+ private transient Thread shutdownThread = null;
Review comment:
is there a reason why this should be `transient`. we dont serialize this, right
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2217: [HUDI-1358] Fix Memory Leak in HoodieLogFormatWriter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2217:
URL: https://github.com/apache/hudi/pull/2217#issuecomment-719081050
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2217?src=pr&el=h1) Report
> Merging [#2217](https://codecov.io/gh/apache/hudi/pull/2217?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/6310a2307abba94c7ff8a770f45462deae2c312e?el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `85.71%`.
[](https://codecov.io/gh/apache/hudi/pull/2217?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2217 +/- ##
============================================
+ Coverage 53.67% 53.71% +0.03%
- Complexity 2849 2851 +2
============================================
Files 359 359
Lines 16565 16571 +6
Branches 1782 1782
============================================
+ Hits 8892 8901 +9
+ Misses 6916 6912 -4
- Partials 757 758 +1
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
| #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| #hudicommon | `54.75% <85.71%> (+0.06%)` | `1796.00 <2.00> (+2.00)` | |
| #hudihadoopmr | `33.05% <ø> (ø)` | `181.00 <ø> (ø)` | |
| #hudispark | `65.95% <ø> (ø)` | `304.00 <ø> (ø)` | |
| #huditimelineservice | `62.29% <ø> (ø)` | `50.00 <ø> (ø)` | |
| #hudiutilities | `70.09% <ø> (ø)` | `327.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2217?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/2217/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `78.81% <85.71%> (+1.13%)` | `23.00 <2.00> (+1.00)` | |
| [...ache/hudi/common/fs/inline/InMemoryFileSystem.java](https://codecov.io/gh/apache/hudi/pull/2217/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL2lubGluZS9Jbk1lbW9yeUZpbGVTeXN0ZW0uamF2YQ==) | `89.65% <0.00%> (+10.34%)` | `16.00% <0.00%> (+1.00%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2217: [HUDI-1358] Fix Memory Leak in HoodieLogFormatWriter
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2217:
URL: https://github.com/apache/hudi/pull/2217#issuecomment-719098832
Please go ahead merge, when ready
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2217: [HUDI-1358] Fix Memory Leak in HoodieLogFormatWriter
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2217:
URL: https://github.com/apache/hudi/pull/2217#issuecomment-719081050
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2217?src=pr&el=h1) Report
> Merging [#2217](https://codecov.io/gh/apache/hudi/pull/2217?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/6310a2307abba94c7ff8a770f45462deae2c312e?el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `85.71%`.
[](https://codecov.io/gh/apache/hudi/pull/2217?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2217 +/- ##
============================================
+ Coverage 53.67% 53.71% +0.03%
- Complexity 2849 2851 +2
============================================
Files 359 359
Lines 16565 16571 +6
Branches 1782 1782
============================================
+ Hits 8892 8901 +9
+ Misses 6916 6912 -4
- Partials 757 758 +1
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
| #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| #hudicommon | `54.75% <85.71%> (+0.06%)` | `1796.00 <2.00> (+2.00)` | |
| #hudihadoopmr | `33.05% <ø> (ø)` | `181.00 <ø> (ø)` | |
| #hudispark | `65.95% <ø> (ø)` | `304.00 <ø> (ø)` | |
| #huditimelineservice | `62.29% <ø> (ø)` | `50.00 <ø> (ø)` | |
| #hudiutilities | `70.09% <ø> (ø)` | `327.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2217?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/2217/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `78.81% <85.71%> (+1.13%)` | `23.00 <2.00> (+1.00)` | |
| [...ache/hudi/common/fs/inline/InMemoryFileSystem.java](https://codecov.io/gh/apache/hudi/pull/2217/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL2lubGluZS9Jbk1lbW9yeUZpbGVTeXN0ZW0uamF2YQ==) | `89.65% <0.00%> (+10.34%)` | `16.00% <0.00%> (+1.00%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #2217: [HUDI-1358] Fix Memory Leak in HoodieLogFormatWriter
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #2217:
URL: https://github.com/apache/hudi/pull/2217#discussion_r520232782
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java
##########
@@ -55,6 +55,8 @@
private final String rolloverLogWriteToken;
private FSDataOutputStream output;
private boolean closed = false;
+ private transient Thread shutdownThread = null;
Review comment:
Yes, This is just to be defensive .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org