You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/07/16 03:29:44 UTC
[GitHub] [incubator-kyuubi] zwangsheng opened a new pull request #815: docs for spark-block-cleaner
zwangsheng opened a new pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815
### _Why are the changes needed?_
Add Docs for kyuubi tools spark-block-cleaner.
* Explain the parameters
* Introduction to basic startup
* Give an example
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881155447
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#815](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (25be318) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/0e5bec78ff9db0346a885e7ee94f538bd5a42e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e5bec7) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #815 +/- ##
=========================================
Coverage 78.70% 78.70%
Complexity 10 10
=========================================
Files 132 132
Lines 5108 5108
Branches 638 638
=========================================
Hits 4020 4020
Misses 740 740
Partials 348 348
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e5bec7...25be318](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670941418
##########
File path: docs/tools/index.rst
##########
@@ -15,3 +15,4 @@ Develop Tools
debugging
community
developer
+ spark_block_cleaner
Review comment:
I'd like add a new entrance like `tools`, then we can put this into `tools`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you edited a comment on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you edited a comment on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881153655
@zwangsheng thank you for working on this, can we put the screen snapshot about this docs ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881155447
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#815](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (25be318) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/0e5bec78ff9db0346a885e7ee94f538bd5a42e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e5bec7) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head 25be318 differs from pull request most recent head f834b38. Consider uploading reports for the commit f834b38 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #815 +/- ##
=========================================
Coverage 78.70% 78.70%
Complexity 10 10
=========================================
Files 132 132
Lines 5108 5108
Branches 638 638
=========================================
Hits 4020 4020
Misses 740 740
Partials 348 348
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e5bec7...f834b38](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881155447
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#815](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7a87f31) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/0e5bec78ff9db0346a885e7ee94f538bd5a42e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e5bec7) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head 7a87f31 differs from pull request most recent head 0803995. Consider uploading reports for the commit 0803995 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #815 +/- ##
=========================================
Coverage 78.70% 78.70%
Complexity 10 10
=========================================
Files 132 132
Lines 5108 5108
Branches 638 638
=========================================
Hits 4020 4020
Misses 740 740
Partials 348 348
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e5bec7...0803995](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881155447
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#815](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7a87f31) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/0e5bec78ff9db0346a885e7ee94f538bd5a42e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e5bec7) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head 7a87f31 differs from pull request most recent head 9cf3e15. Consider uploading reports for the commit 9cf3e15 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #815 +/- ##
=========================================
Coverage 78.70% 78.70%
Complexity 10 10
=========================================
Files 132 132
Lines 5108 5108
Branches 638 638
=========================================
Hits 4020 4020
Misses 740 740
Partials 348 348
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e5bec7...9cf3e15](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670942229
##########
File path: docs/tools/spark_block_cleaner.md
##########
@@ -0,0 +1,97 @@
+<div align=center>
+
+
+
+</div>
+
+# Kubernetes tools spark-block-cleaner
+
+## Requirements
+You'd better have cognition upon the following things when you want to use spark-block-cleaner.
+
+* Read this article
+* An active Kubernetes cluster
+* [Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
+
+## Purpose
+
+When running Spark On Kubernetes, we encountered such a situation that after using hostPath volume local-dir, the usage rate of the directory storing shuffle files remained high.
+
+So an additional tool is needed to clear the accumulated block files.
+
+## Usage
+
+### Build Block Cleaner Docker Image
+```shell
+ docker build ./tools/spark-block-cleaner/kubernetes/docker
+```
+
+### Modify spark-block-cleaner.yml
+You need to modify the spark-block-cleaner.yml to fit your current environment.
Review comment:
`spark-block-cleaner.yml` -> `**spark-block-cleaner.yml**`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881155447
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#815](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1ec6795) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/0e5bec78ff9db0346a885e7ee94f538bd5a42e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e5bec7) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #815 +/- ##
=========================================
Coverage 78.70% 78.70%
Complexity 10 10
=========================================
Files 132 132
Lines 5108 5108
Branches 638 638
=========================================
Hits 4020 4020
Misses 740 740
Partials 348 348
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e5bec7...1ec6795](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670941831
##########
File path: docs/tools/spark_block_cleaner.md
##########
@@ -0,0 +1,97 @@
+<div align=center>
+
+
+
+</div>
+
+# Kubernetes tools spark-block-cleaner
+
+## Requirements
+You'd better have cognition upon the following things when you want to use spark-block-cleaner.
+
+* Read this article
+* An active Kubernetes cluster
+* [Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
+
+## Purpose
+
+When running Spark On Kubernetes, we encountered such a situation that after using hostPath volume local-dir, the usage rate of the directory storing shuffle files remained high.
+
+So an additional tool is needed to clear the accumulated block files.
Review comment:
`clear ` -> `clean up`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881155447
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#815](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bbf4d6e) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/0e5bec78ff9db0346a885e7ee94f538bd5a42e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e5bec7) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head bbf4d6e differs from pull request most recent head 1ec6795. Consider uploading reports for the commit 1ec6795 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #815 +/- ##
=========================================
Coverage 78.70% 78.70%
Complexity 10 10
=========================================
Files 132 132
Lines 5108 5108
Branches 638 638
=========================================
Hits 4020 4020
Misses 740 740
Partials 348 348
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e5bec7...1ec6795](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670942229
##########
File path: docs/tools/spark_block_cleaner.md
##########
@@ -0,0 +1,97 @@
+<div align=center>
+
+
+
+</div>
+
+# Kubernetes tools spark-block-cleaner
+
+## Requirements
+You'd better have cognition upon the following things when you want to use spark-block-cleaner.
+
+* Read this article
+* An active Kubernetes cluster
+* [Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
+
+## Purpose
+
+When running Spark On Kubernetes, we encountered such a situation that after using hostPath volume local-dir, the usage rate of the directory storing shuffle files remained high.
+
+So an additional tool is needed to clear the accumulated block files.
+
+## Usage
+
+### Build Block Cleaner Docker Image
+```shell
+ docker build ./tools/spark-block-cleaner/kubernetes/docker
+```
+
+### Modify spark-block-cleaner.yml
+You need to modify the spark-block-cleaner.yml to fit your current environment.
Review comment:
**spark-block-cleaner.yml**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you closed pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881155447
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#815](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0803995) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/0e5bec78ff9db0346a885e7ee94f538bd5a42e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e5bec7) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head 0803995 differs from pull request most recent head 9cf3e15. Consider uploading reports for the commit 9cf3e15 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #815 +/- ##
=========================================
Coverage 78.70% 78.70%
Complexity 10 10
=========================================
Files 132 132
Lines 5108 5108
Branches 638 638
=========================================
Hits 4020 4020
Misses 740 740
Partials 348 348
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e5bec7...9cf3e15](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zwangsheng commented on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
zwangsheng commented on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881232796
Straighten out the article，cc @ulysses-you @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zwangsheng commented on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
zwangsheng commented on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881150183
cc @ulysses-you @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zwangsheng commented on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
zwangsheng commented on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881360650
> Have you addressed this comment [#815 (comment)](https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670947998) ?
yes, offline communication. It is ok to use `develop tools` and `tools`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670942625
##########
File path: docs/tools/spark_block_cleaner.md
##########
@@ -0,0 +1,97 @@
+<div align=center>
+
+
+
+</div>
+
+# Kubernetes tools spark-block-cleaner
+
+## Requirements
+You'd better have cognition upon the following things when you want to use spark-block-cleaner.
+
+* Read this article
+* An active Kubernetes cluster
+* [Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
+
+## Purpose
+
+When running Spark On Kubernetes, we encountered such a situation that after using hostPath volume local-dir, the usage rate of the directory storing shuffle files remained high.
+
+So an additional tool is needed to clear the accumulated block files.
+
+## Usage
+
+### Build Block Cleaner Docker Image
+```shell
+ docker build ./tools/spark-block-cleaner/kubernetes/docker
+```
+
+### Modify spark-block-cleaner.yml
+You need to modify the spark-block-cleaner.yml to fit your current environment.
+
+```yaml
+volumes:
+ # Directory on the host which store block dirs
+ - name: block-files-dir-1
+ hostPath:
+ path: /blockFilesDirs/data1
+ - name: block-files-dir-2
+ hostPath:
+ path: /blockFilesDirs/data2
+ # Directory on the host which you want to store clean log
+ - name: cleaner-log
+ hostPath:
+ path: /logDir
+```
+
+The above parameters used to help volume hostPath to container path.
+
+Then you should modify following parameter to fit environment.
+
+The `CACHE_DIRS` value should fit your actual situation.
+
+```yaml
+env:
+ # Set env to manager cleaner running
+ # the target dirs which in container
+ - name: CACHE_DIRS
+ value: /data/data1,/data/data2
+```
+
+Such as you used /blockFilesDirs/data1, /blockFilesDirs/data2 as spark-local-dirs to cache shuffle data.
+
+You should modify
+```yaml
+volumes:
+ - name: block-files-dir-1
+ hostPath:
+ path: /blockFilesDirs/data1
+ - name: block-files-dir-2
+ hostPath:
+ path: /blockFilesDirs/data2
+```
+
+And then
+
+```yaml
+env:
+ # Set env to manager cleaner running
+ # the target dirs which in container
+ - name: CACHE_DIRS
+ value: /data/data1,/data/data2
+```
+You can modify the following parameters in spark-block-cleaner.yml containers envs to control spark-block-cleaner running.
+
+### Start daemonSet
+
+After you finishing modifying the above, you can use the following command `kubectl apply -f spark-block-cleaner.yml` to start daemonSet.
+
+## Related parameters
+
+Name | Default | Meaning
+--- | --- | ---
+CACHE_DIRS | /data/data1,/data/data2 | The target dirs in container path which will clean block files.
+FILE_EXPIRED_TIME | 604800 | Cleaner will clean the block files which current time - last modified time more than the fileExpiredTime.
+DEEP_CLEAN_FILE_EXPIRED_TIME | 432000 | Deep clean will clean the block files which current time - last modified time more than the deepCleanFileExpiredTime.
+FREE_SPACE_THRESHOLD | 60 | After first clean, if free Space low than threshold trigger deep clean.
+SCHEDULE_INTERVAL | 3600 | Cleaner sleep between cleaning.
Review comment:
add the time unit for these config (seconds).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zwangsheng commented on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
zwangsheng commented on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881156232
@ulysses-you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881153655
@zwangsheng thank you for working on this, can we put the snapshot about this docs ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881155447
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#815](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7a87f31) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/0e5bec78ff9db0346a885e7ee94f538bd5a42e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e5bec7) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head 7a87f31 differs from pull request most recent head f834b38. Consider uploading reports for the commit f834b38 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #815 +/- ##
=========================================
Coverage 78.70% 78.70%
Complexity 10 10
=========================================
Files 132 132
Lines 5108 5108
Branches 638 638
=========================================
Hits 4020 4020
Misses 740 740
Partials 348 348
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e5bec7...f834b38](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670947998
##########
File path: docs/tools/index.rst
##########
@@ -1,17 +1,11 @@
.. image:: ../imgs/kyuubi_logo.png
:align: center
-Develop Tools
+Tools
Review comment:
Auxiliaries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zwangsheng commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
zwangsheng commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670941910
##########
File path: docs/tools/index.rst
##########
@@ -15,3 +15,4 @@ Develop Tools
debugging
community
developer
+ spark_block_cleaner
Review comment:
Good suggestion, i will move this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881333542
Have you addressed this comment https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670947998 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881486404
thanks, merging to master for v1.3.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r671044026
##########
File path: docs/tools/spark_block_cleaner.md
##########
@@ -4,90 +4,105 @@
</div>
-# Kubernetes tools spark-block-cleaner
+# Kubernetes Tools Spark Block Cleaner
## Requirements
You'd better have cognition upon the following things when you want to use spark-block-cleaner.
* Read this article
* An active Kubernetes cluster
* [Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
+* [Docker](https://www.docker.com/)
## Purpose
+During using Spark On Kubernetes with client deploy-mode, we encountered this scenario that the disk responsible for storing Shuffle data accumulates so many files cause the disk overflows.
Review comment:
Do not tell ‘our’ story, say it as a common case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zwangsheng commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
zwangsheng commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r671047145
##########
File path: docs/tools/spark_block_cleaner.md
##########
@@ -4,90 +4,105 @@
</div>
-# Kubernetes tools spark-block-cleaner
+# Kubernetes Tools Spark Block Cleaner
## Requirements
You'd better have cognition upon the following things when you want to use spark-block-cleaner.
* Read this article
* An active Kubernetes cluster
* [Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
+* [Docker](https://www.docker.com/)
## Purpose
+During using Spark On Kubernetes with client deploy-mode, we encountered this scenario that the disk responsible for storing Shuffle data accumulates so many files cause the disk overflows.
Review comment:
Ok，I will make it common
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r671046917
##########
File path: docs/tools/spark_block_cleaner.md
##########
@@ -4,90 +4,105 @@
</div>
-# Kubernetes tools spark-block-cleaner
+# Kubernetes Tools Spark Block Cleaner
## Requirements
You'd better have cognition upon the following things when you want to use spark-block-cleaner.
* Read this article
* An active Kubernetes cluster
* [Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
+* [Docker](https://www.docker.com/)
## Purpose
+During using Spark On Kubernetes with client deploy-mode, we encountered this scenario that the disk responsible for storing Shuffle data accumulates so many files cause the disk overflows.
-When running Spark On Kubernetes, we encountered such a situation that after using hostPath volume local-dir, the usage rate of the directory storing shuffle files remained high.
+Therefore, we chose to use Spark Block Cleaner to clear the block files accumulated by Spark.
-So an additional tool is needed to clean up the accumulated block files.
+## Scenes
+When you're using Spark On Kubernetes with Client mode and don't use `emptyDir` for Spark `local-dir` type, you may face the same scenario that executor pods deleted without clean all the Block files.
+
+## Principle
+When deploying Spark Block Cleaner, we will configure volumes for the destination folder. Spark Block Cleaner will perceive the folder by the parameter `CACHE_DIRS`.
+
+Spark Block Cleaner will clear the perceived folder in a fixed loop(which can be configured by `SCHEDULE_INTERVAL`). And Spark Block Cleaner will select folder start with `blockmgr` and `spark` for deletion using the logic Spark uses to create those folders.
+
+Before deleting those files, Spark Block Cleaner will determine whether it is a recently modified file(depending on whether the file has not been acted on within the specified time which configured by `FILE_EXPIRED_TIME`). Only delete files those beyond that time interval.
+
+And Spark Block Cleaner will check the disk utilization after clean, if the remaining space is less than the specified value(control by `FREE_SPACE_THRESHOLD`), will trigger deep clean(which file expired time control by `DEEP_CLEAN_FILE_EXPIRED_TIME`).
## Usage
+Before you start using Spark Block Cleaner, you should build its docker images or using official images(TODO).
Review comment:
~~or using official images(TODO)~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #815: [DOC] [KUBERNETES] Doc for spark-block-cleaner
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #815:
URL: https://github.com/apache/incubator-kyuubi/pull/815#issuecomment-881155447
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#815](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9cf3e15) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/0e5bec78ff9db0346a885e7ee94f538bd5a42e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e5bec7) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #815 +/- ##
============================================
+ Coverage 78.70% 78.71% +0.01%
Complexity 10 10
============================================
Files 132 132
Lines 5108 5108
Branches 638 638
============================================
+ Hits 4020 4021 +1
Misses 740 740
+ Partials 348 347 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/kyuubi/sql/KyuubiQueryStagePreparation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/815/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmtfMy4xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9LeXV1YmlRdWVyeVN0YWdlUHJlcGFyYXRpb24uc2NhbGE=) | `80.39% <0.00%> (+0.98%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e5bec7...9cf3e15](https://codecov.io/gh/apache/incubator-kyuubi/pull/815?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org