You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/03/19 16:27:08 UTC
[GitHub] [lucene] mocobeta opened a new pull request #755: Add note for --tmp-dir option in releaseWizard
mocobeta opened a new pull request #755:
URL: https://github.com/apache/lucene/pull/755
I encountered several times a "disk full" error when running the smoke tester.
From my observation, there is a test that (intentionally) creates very large index files. It might be helpful to have a notice about that in the rc announcing mails and encourage to explicitly use `--tmp-dir` option to avoid unnecessary retries.
Here is a snapshot of the disk usage in a smoke tester run.
```
$ dust -b ~/tmp/smoketester/
47M ┌── lucene-9.1.0-src.tgz
64M ├── lucene-9.1.0.tgz
37M │ ┌── test-framework
59M │ │ ┌── lucene-9.1.0-SNAPSHOT-itests
59M │ │ ┌─┴ packages
59M │ │ ┌─┴ build
59M │ ├─┴ distribution
51M │ │ ┌── build
63M │ ├─┴ backward-codecs
125M │ │ ┌── site
125M │ │ ┌─┴ build
125M │ ├─┴ documentation
55M │ │ ┌── nori
133M │ ├─┴ analysis
3.0G │ │ ┌── _0.cfs
4.0G │ │ ├── _0.fdt
7.1G │ │ ┌─┴ 4GBStoredFields-001
7.1G │ │ ┌─┴ lucene.index.Test4GBStoredFields_1FCE4B52E7A84AF3-001
7.1G │ │ ┌─┴ tests-tmp
7.1G │ │ ┌─┴ tmp
7.1G │ │ ┌─┴ build
7.1G │ ├─┴ core
7.6G │ ┌─┴ lucene
7.6G │ ┌─┴ lucene-9.1.0
7.6G ├─┴ unpack
7.7G ┌─┴ smoketester
```
I didn't (couldn't) test this but just grepped the scripts to find where to change; would you tell me the correct file if I'm modifying the wrong file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] msokolov commented on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
msokolov commented on pull request #755:
URL: https://github.com/apache/lucene/pull/755#issuecomment-1073041616
Big plus one for this. I can never complete the smoke test without it and I
always forget the parameter name and where it has to go on the command
line. I don't think it can be tacked on the end
On Sat, Mar 19, 2022, 12:27 PM Tomoko Uchida ***@***.***>
wrote:
> I encountered several times a "disk full" error when running the smoke
> tester.
> From my observation, there is a test that (intentionally) creates very
> large index files. It might be helpful to have a notice about that in the
> rc announcing mails and encourage to explicitly use --tmp-dir option to
> avoid unnecessary retries.
>
> Here is a snapshot of the disk usage in a smoke tester run.
>
> $ dust -b ~/tmp/smoketester/
>
> 47M ┌── lucene-9.1.0-src.tgz
>
> 64M ├── lucene-9.1.0.tgz
>
> 37M │ ┌── test-framework
>
> 59M │ │ ┌── lucene-9.1.0-SNAPSHOT-itests
>
> 59M │ │ ┌─┴ packages
>
> 59M │ │ ┌─┴ build
>
> 59M │ ├─┴ distribution
>
> 51M │ │ ┌── build
>
> 63M │ ├─┴ backward-codecs
>
> 125M │ │ ┌── site
>
> 125M │ │ ┌─┴ build
>
> 125M │ ├─┴ documentation
>
> 55M │ │ ┌── nori
>
> 133M │ ├─┴ analysis
>
> 3.0G │ │ ┌── _0.cfs
>
> 4.0G │ │ ├── _0.fdt
>
> 7.1G │ │ ┌─┴ 4GBStoredFields-001
>
> 7.1G │ │ ┌─┴ lucene.index.Test4GBStoredFields_1FCE4B52E7A84AF3-001
>
> 7.1G │ │ ┌─┴ tests-tmp
>
> 7.1G │ │ ┌─┴ tmp
>
> 7.1G │ │ ┌─┴ build
>
> 7.1G │ ├─┴ core
>
> 7.6G │ ┌─┴ lucene
>
> 7.6G │ ┌─┴ lucene-9.1.0
>
> 7.6G ├─┴ unpack
>
> 7.7G ┌─┴ smoketester
>
>
> I didn't (couldn't) test this but just grepped the scripts to find where
> to change; would you tell me the correct file if I'm modifying the wrong
> file.
> ------------------------------
> You can view, comment on, or merge this pull request online at:
>
> https://github.com/apache/lucene/pull/755
> Commit Summary
>
> - b633070
> <https://github.com/apache/lucene/pull/755/commits/b6330704b26410c7ffb535e85dcf31d1f8b3d62a>
> add note for --tmp-dir option
>
> File Changes
>
> (1 file <https://github.com/apache/lucene/pull/755/files>)
>
> - *M* dev-tools/scripts/releaseWizard.yaml
> <https://github.com/apache/lucene/pull/755/files#diff-a28a83d173444b2a99df45d0bd2f0e77f3942a296259c4f66a56b932d8ae0e99>
> (4)
>
> Patch Links:
>
> - https://github.com/apache/lucene/pull/755.patch
> - https://github.com/apache/lucene/pull/755.diff
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/lucene/pull/755>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHHUQLWJ2YLIZPXTSWXU4LVAX55TANCNFSM5REG6KAA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
jpountz commented on pull request #755:
URL: https://github.com/apache/lucene/pull/755#issuecomment-1073690606
One downside of this approach is that nobody could just copy-paste the suggested command to run the smoke tester any longer, they would always have to tune the temporary path. I'd rather turn `Test4GBStoredFields` into a `@Monster` test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
mocobeta commented on pull request #755:
URL: https://github.com/apache/lucene/pull/755#issuecomment-1073452732
@janhoy Would you take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mocobeta edited a comment on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
mocobeta edited a comment on pull request #755:
URL: https://github.com/apache/lucene/pull/755#issuecomment-1073827624
Thanks for your feedback and suggestions.
I reverted the command line to keep it as is.
The option could still be useful for some situations, I added a description of it in the notice below the command. Does it make sense?
> I'd rather turn Test4GBStoredFields into a @ Monster test?
Maybe an issue is needed to change the test, I will open it.
> Perhaps we could have an env var LUCENE_SMOKE_TESTER_TMP_DIR that the smoke tester would respect, then devs can set that once and be happy?
It would be possible to make the smoke tester respect an env var if it is set as the default, and people will have options to change the location (an env var or explicit `--tmp-dir` option; the latter takes priority). I think this change can be done in this PR rather than opening additional one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
mocobeta commented on pull request #755:
URL: https://github.com/apache/lucene/pull/755#issuecomment-1074927120
The test in question was excluded from nightly tests (LUCENE-10478). We may not need to change the smoke tester or release wizard for disk usage - if a test uses lots of disk space, it shouldn't be run in the smoke tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
mocobeta commented on pull request #755:
URL: https://github.com/apache/lucene/pull/755#issuecomment-1073174420
My `tmpfs` partition (where system `/tmp` is mounted) has 8GB of available disk space.
The test itself has been there for a long time and I think I had not seen this "disk full" error before 9.0. The disk usage could have increased since 9.x for some reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mocobeta closed pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
mocobeta closed pull request #755:
URL: https://github.com/apache/lucene/pull/755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mocobeta edited a comment on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
mocobeta edited a comment on pull request #755:
URL: https://github.com/apache/lucene/pull/755#issuecomment-1073174420
My `tmpfs` partition (where system `/tmp` is mounted) has 8GB of available disk space.
The test itself has been there for a long time and I think I had not seen this "disk full" error before 9.0. The disk usage in tests could have increased since 9.x for some reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #755:
URL: https://github.com/apache/lucene/pull/755#issuecomment-1073746350
Perhaps we could have an env var `LUCENE_SMOKE_TESTER_TMP_DIR` that the smoke tester would respect, then devs can set that once and be happy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mocobeta commented on pull request #755: Add note for smoke tester --tmp-dir option in rc announcing
Posted by GitBox <gi...@apache.org>.
mocobeta commented on pull request #755:
URL: https://github.com/apache/lucene/pull/755#issuecomment-1073827624
Thanks for your feedback and suggestions.
I reverted the command line to keep it as is.
The option could still be useful for some situations, I added a description of it in the notice below the command. Does it make sense?
> I'd rather turn Test4GBStoredFields into a @ Monster test?
Maybe an issue is needed to change the test, I will open it.
> Perhaps we could have an env var LUCENE_SMOKE_TESTER_TMP_DIR that the smoke tester would respect, then devs can set that once and be happy?
It would be possible to make the smoke tester respect an env var if it is set as the default, and people will have options to change the location (an env var or explicit `--tmp-dir` option; the latter overwrite the env var). I think this change can be done in this PR rather than opening additional one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org