You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/25 18:24:26 UTC
[GitHub] [beam] nevillelyh opened a new pull request #12681: reduce disk usage in NativeFileSorter
nevillelyh opened a new pull request #12681:
URL: https://github.com/apache/beam/pull/12681
`dataFile` is used append only and redundant as soon as `sortInBatch`
splits it into batch sorted files
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace
--- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] stale[bot] commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-751236018
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-797064646
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles edited a comment on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
kennknowles edited a comment on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-797066840
Pulled this PR and ran it `./gradlew :sdks:java:extensions:sorter:spotbugsMain`
```
Exceptional return value of java.io.File.delete() ignored in org.apache.beam.sdk.extensions.sorter.NativeFileSorter.sortInBatch()
Bug type RV_RETURN_VALUE_IGNORED_BAD_PRACTICE (click for details)
In class org.apache.beam.sdk.extensions.sorter.NativeFileSorter
In method org.apache.beam.sdk.extensions.sorter.NativeFileSorter.sortInBatch()
Called method java.io.File.delete()
At NativeFileSorter.java:[line 142]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-756455027
All builds look green. Am I missing a build?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-784519144
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-756366028
What is the next step on this PR? Is this stale? Could it be merged?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-785470907
Last version is failing with
```
18:07:27 Execution failed for task ':sdks:java:extensions:sorter:spotbugsMain'.
18:07:27 > A failure occurred while executing com.github.spotbugs.snom.internal.SpotBugsRunnerForWorker$SpotBugsExecutor
18:07:27 > 1 SpotBugs violations were found. See the report at: file:///home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Phrase/src/sdks/java/extensions/sorter/build/reports/spotbugs/main.xml
```
@kennknowles - Do you know how to resolve this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-797058046
> Last version is failing with
>
> ```
> 18:07:27 Execution failed for task ':sdks:java:extensions:sorter:spotbugsMain'.
> 18:07:27 > A failure occurred while executing com.github.spotbugs.snom.internal.SpotBugsRunnerForWorker$SpotBugsExecutor
> 18:07:27 > 1 SpotBugs violations were found. See the report at: file:///home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Phrase/src/sdks/java/extensions/sorter/build/reports/spotbugs/main.xml
> ```
>
> @kennknowles - Do you know how to resolve this?
@kennknowles - ping?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nevillelyh commented on a change in pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
nevillelyh commented on a change in pull request #12681:
URL: https://github.com/apache/beam/pull/12681#discussion_r578852503
##########
File path: sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java
##########
@@ -139,6 +139,7 @@ public void add(byte[] key, byte[] value) throws IOException {
}
} finally {
inputStream.close();
+ Preconditions.checkArgument(dataFile.delete());
Review comment:
```suggestion
dataFile.delete();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nevillelyh commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
nevillelyh commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-756367459
It's ready to merge. No sure why builds are failing though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #12681:
URL: https://github.com/apache/beam/pull/12681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-784675761
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a change in pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
kennknowles commented on a change in pull request #12681:
URL: https://github.com/apache/beam/pull/12681#discussion_r592741168
##########
File path: sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java
##########
@@ -139,6 +139,7 @@ public void add(byte[] key, byte[] value) throws IOException {
}
} finally {
inputStream.close();
+ dataFile.delete();
Review comment:
So spotbugs is suggesting that you ought to do something like
```
if (!dataFile.delete()) {
// Log a warning? Throw an exception?
}
```
##########
File path: sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java
##########
@@ -139,6 +139,7 @@ public void add(byte[] key, byte[] value) throws IOException {
}
} finally {
inputStream.close();
+ dataFile.delete();
Review comment:
So spotbugs is suggesting that you ought to do something like
```java
if (!dataFile.delete()) {
// Log a warning? Throw an exception?
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #12681:
URL: https://github.com/apache/beam/pull/12681#discussion_r554088688
##########
File path: sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java
##########
@@ -139,6 +139,7 @@ public void add(byte[] key, byte[] value) throws IOException {
}
} finally {
inputStream.close();
+ Preconditions.checkArgument(dataFile.delete());
Review comment:
It was not obvious to me initially. IIUC, this is safe only because `sort` could only be called once.
One question for my learning: Is not the primary purpose of Preconditions is to check input arguments to a function? In this case it is used to assert on the return value of delete call, and that is not a precondition here. Even though functionally this does not make a difference. (And a related question, do we even need to check this? Even if delete fails here that is ok, it is a temp file and will be cleaned on exit.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #12681:
URL: https://github.com/apache/beam/pull/12681#discussion_r554088688
##########
File path: sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java
##########
@@ -139,6 +139,7 @@ public void add(byte[] key, byte[] value) throws IOException {
}
} finally {
inputStream.close();
+ Preconditions.checkArgument(dataFile.delete());
Review comment:
It was not obvious to me initially. IIUC, this is safe only because `sort` could only be called once.
One question for my learning: Is not the primary purpose of Preconditions is to check input arguments to a function? In this case it is used to assert on the return value of delete call, and that is not a precondition here. Even though functionally this does not make a difference. (And a related question, do we even need to check this? Even if delete fails here that is ok, it is a temp file and will be cleaned on exit.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-1041455048
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-1032573407
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-797066840
Pulled this PR and ran it `./gradlew :sdks:java:extensions:sorter:spotbugsMain`
```
Exceptional return value of java.io.File.delete() ignored in org.apache.beam.sdk.extensions.sorter.NativeFileSorter.sortInBatch()
--
| Bug type RV_RETURN_VALUE_IGNORED_BAD_PRACTICE (click for details)In class org.apache.beam.sdk.extensions.sorter.NativeFileSorterIn method org.apache.beam.sdk.extensions.sorter.NativeFileSorter.sortInBatch()Called method java.io.File.delete()At NativeFileSorter.java:[line 142]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nevillelyh commented on pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
nevillelyh commented on pull request #12681:
URL: https://github.com/apache/beam/pull/12681#issuecomment-756460061
Never mind I was looking at "Post-Commit Tests Status (on master branch)".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nevillelyh commented on a change in pull request #12681: reduce disk usage in NativeFileSorter
Posted by GitBox <gi...@apache.org>.
nevillelyh commented on a change in pull request #12681:
URL: https://github.com/apache/beam/pull/12681#discussion_r578852703
##########
File path: sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java
##########
@@ -139,6 +139,7 @@ public void add(byte[] key, byte[] value) throws IOException {
}
} finally {
inputStream.close();
+ Preconditions.checkArgument(dataFile.delete());
Review comment:
You're right. No idea why I had `checkArgument` here. Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org