You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mark Miller (JIRA)" <ji...@apache.org> on 2009/08/01 22:24:14 UTC
[jira] Created: (LUCENE-1774) In a benchmark alg, if you use a
log.step thats higher than the number of docs you add, whacky high numbers
are logged for doc adds
In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
-----------------------------------------------------------------------------------------------------------------------------------
Key: LUCENE-1774
URL: https://issues.apache.org/jira/browse/LUCENE-1774
Project: Lucene - Java
Issue Type: Bug
Components: contrib/benchmark
Reporter: Mark Miller
Priority: Trivial
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1774) In a benchmark alg, if you use a
log.step thats higher than the number of docs you add, whacky high numbers
are logged for doc adds
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738332#action_12738332 ]
Mark Miller commented on LUCENE-1774:
-------------------------------------
Good idea!
I'm not so keen on adding it as a param though.
What about using log.step as a default that works for all tasks, and also having log.step.PerfTask that can control by task?
Then delete would be log.step.DeleteDocTask=100 and you could individually set any of the others if you choose.
Or Something along those lines? I almost think the setting should stay out of the alg logic though - but it would be nice to have finer control - to do things like match the old behavior:
log.step.AddDocTask=1000
Def deserves a new issue if we do something here I think.
> In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1774
> URL: https://issues.apache.org/jira/browse/LUCENE-1774
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Mark Miller
> Priority: Trivial
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1774) In a benchmark alg, if you use a
log.step thats higher than the number of docs you add, whacky high numbers
are logged for doc adds
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738346#action_12738346 ]
Shai Erera commented on LUCENE-1774:
------------------------------------
I like that idea. DeleteDocTask today defines a "delete.log.step", but it's defined only for that task. There is no "search.log.step" for example.
Having a pattern like log.step.<Task Class name> will allow us to support it in PerfTask by doing something like:
{code}
logStep = config.get("log.step", DEFAULT_LOG_STEP);
if (logStep <= 0) {
logStep = Integer.MAX_VALUE;
}
// Check if a log step for the current task is defined
logStep = config.get("log.step" + getClass().getSimpleName(), DEFAULT_LOG_STEP);
if (logStep <= 0) {
logStep = Integer.MAX_VALUE;
}
{code}
And so we can support a log.step for any task, irregardless of whether that task included support for it or not. And we can remove the delete.log.step param (new in 2.9 so no problem). Note that the above code is 1.5 dependent, but I can make it 1.4 compliant w/ a TODO to switch to getSimpleName() in 1.5.
What do you think? Shall I open an issue for that? Or can you reopen that issue and I'll prepare a patch?
> In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1774
> URL: https://issues.apache.org/jira/browse/LUCENE-1774
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Mark Miller
> Priority: Trivial
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1774) In a benchmark alg, if you use a
log.step thats higher than the number of docs you add, whacky high numbers
are logged for doc adds
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738077#action_12738077 ]
Mark Miller commented on LUCENE-1774:
-------------------------------------
Yeah, I'll post some further info when I get a moment - it may have been some aberration or something? I was seeing it while working with some enwiki benchmark stuff yesterday - if I put a step of 10,000 and then added 500 docs, it would output, 10,000, 20,000, 30,000 - 50,000 for every 100. I'll try and duplicate it again when I get a chance.
> In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1774
> URL: https://issues.apache.org/jira/browse/LUCENE-1774
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Mark Miller
> Priority: Trivial
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1774) In a benchmark alg, if you use a
log.step thats higher than the number of docs you add, whacky high numbers
are logged for doc adds
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738349#action_12738349 ]
Mark Miller commented on LUCENE-1774:
-------------------------------------
bq. What do you think? Shall I open an issue for that? Or can you reopen that issue and I'll prepare a patch?
+1 - I vote new issue for it. If you make the patch, I'll commit it for 2.9.
> In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1774
> URL: https://issues.apache.org/jira/browse/LUCENE-1774
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Mark Miller
> Priority: Trivial
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1774) In a benchmark alg, if you use a
log.step thats higher than the number of docs you add, whacky high numbers
are logged for doc adds
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738090#action_12738090 ]
Shai Erera commented on LUCENE-1774:
------------------------------------
strange. I've tried to reproduce but failed, so if you succeed, I'd like to take a look too.
> In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1774
> URL: https://issues.apache.org/jira/browse/LUCENE-1774
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Mark Miller
> Priority: Trivial
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1774) In a benchmark alg, if you use a
log.step thats higher than the number of docs you add, whacky high numbers
are logged for doc adds
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738033#action_12738033 ]
Shai Erera commented on LUCENE-1774:
------------------------------------
Mark - can you perhaps paste here an .alg file and the output? I executed readContentSource.alg like this, but didn't see anything "whacky" printed. Perhaps I'm not running the exact algorithm? Does it have to do w/ indexing?
{code}
content.source=org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource
docs.file=temp/enwiki-20070527-pages-articles.xml.bz2
# Stop after processing the document feed once:
content.source.forever=false
# Log messages every:
log.step=100000
# -------------------------------------------------------------------------------------
{ ConsumeContentSource } : 10
RepSumByPref ConsumeContentSource
{code}
I also tried indexLineFile.alg, but again I didn't see anything weird.
> In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1774
> URL: https://issues.apache.org/jira/browse/LUCENE-1774
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Mark Miller
> Priority: Trivial
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-1774) In a benchmark alg, if
you use a log.step thats higher than the number of docs you add, whacky
high numbers are logged for doc adds
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738346#action_12738346 ]
Shai Erera edited comment on LUCENE-1774 at 8/3/09 6:55 AM:
------------------------------------------------------------
I like that idea. DeleteDocTask today defines a "delete.log.step", but it's defined only for that task. There is no "search.log.step" for example.
Having a pattern like log.step.<Task Class name> will allow us to support it in PerfTask by doing something like:
{code}
logStep = config.get("log.step", DEFAULT_LOG_STEP);
if (logStep <= 0) {
logStep = Integer.MAX_VALUE;
}
// Check if a log step for the current task is defined
logStep = config.get("log.step." + getClass().getSimpleName(), DEFAULT_LOG_STEP);
if (logStep <= 0) {
logStep = Integer.MAX_VALUE;
}
{code}
And so we can support a log.step for any task, irregardless of whether that task included support for it or not. And we can remove the delete.log.step param (new in 2.9 so no problem). Note that the above code is 1.5 dependent, but I can make it 1.4 compliant w/ a TODO to switch to getSimpleName() in 1.5.
What do you think? Shall I open an issue for that? Or can you reopen that issue and I'll prepare a patch?
was (Author: shaie):
I like that idea. DeleteDocTask today defines a "delete.log.step", but it's defined only for that task. There is no "search.log.step" for example.
Having a pattern like log.step.<Task Class name> will allow us to support it in PerfTask by doing something like:
{code}
logStep = config.get("log.step", DEFAULT_LOG_STEP);
if (logStep <= 0) {
logStep = Integer.MAX_VALUE;
}
// Check if a log step for the current task is defined
logStep = config.get("log.step" + getClass().getSimpleName(), DEFAULT_LOG_STEP);
if (logStep <= 0) {
logStep = Integer.MAX_VALUE;
}
{code}
And so we can support a log.step for any task, irregardless of whether that task included support for it or not. And we can remove the delete.log.step param (new in 2.9 so no problem). Note that the above code is 1.5 dependent, but I can make it 1.4 compliant w/ a TODO to switch to getSimpleName() in 1.5.
What do you think? Shall I open an issue for that? Or can you reopen that issue and I'll prepare a patch?
> In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1774
> URL: https://issues.apache.org/jira/browse/LUCENE-1774
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Mark Miller
> Priority: Trivial
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1774) In a benchmark alg, if you use a
log.step thats higher than the number of docs you add, whacky high numbers
are logged for doc adds
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738109#action_12738109 ]
Shai Erera commented on LUCENE-1774:
------------------------------------
That is indeed new (I added it in LUCENE-1595 as part of the general refactoring). Now PerfTask includes logging logic, determined by log.step. It means that one log.step controls all tasks, which I'm not sure if it's good or not (doc.delete.log.step controls just deletes because it seemed to be used by several algs).
Maybe, as part of this issue or another issue, we can have PerfTask support a parameter, which will be the log.step? That way you can choose to AddDocTask(100), and SearchTask(-1) to enable logging for adding docs and disable for search. Though that will conflict w/ the other parameter AddDocTask supports (document size). So perhaps we can enhance the parameters to accept a list? Just a thought.
> In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1774
> URL: https://issues.apache.org/jira/browse/LUCENE-1774
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Mark Miller
> Priority: Trivial
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1774) In a benchmark alg, if you use a
log.step thats higher than the number of docs you add, whacky high numbers
are logged for doc adds
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller resolved LUCENE-1774.
---------------------------------
Resolution: Invalid
My fault - had 50k search task that was doing the logging. I don't know if thats new or not, but I was used to just seeing that with adding docs - so when it still jumped by 10,000 when i added 500 docs it appeared a bit weird - when I lowered the step to 100, I saw what I expected - but I didn't realize that the odd numbers I was seeing was the searching task logging by log.step.
> In a benchmark alg, if you use a log.step thats higher than the number of docs you add, whacky high numbers are logged for doc adds
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1774
> URL: https://issues.apache.org/jira/browse/LUCENE-1774
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Mark Miller
> Priority: Trivial
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org