You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org> on 2008/07/09 19:31:31 UTC
[jira] Created: (SOLR-624) patch: Don't take snapshot if there are
no differences
patch: Don't take snapshot if there are no differences
------------------------------------------------------
Key: SOLR-624
URL: https://issues.apache.org/jira/browse/SOLR-624
Project: Solr
Issue Type: Improvement
Components: replication
Affects Versions: 1.3
Reporter: Richard "Trey" Hyde
Fix For: 1.3
This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-624) patch: Don't take snapshot if there
are no differences
Posted by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612570#action_12612570 ]
Richard "Trey" Hyde commented on SOLR-624:
------------------------------------------
My copy of snapcleaner (-N option) from a few weeks ago in 1.3 is also doing the snap* thing, you might want to look into that one too.
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Fix For: 1.3
>
> Attachments: solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-624) patch: Don't take snapshot if there
are no differences
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612374#action_12612374 ]
Hoss Man commented on SOLR-624:
-------------------------------
FWIW: issues with the amount of ram/swap needed in order to fork&exec the replication scripts from java is another reason i know some people cron snapshooter.
short circuiting in the NOOP case out is rarely a bad idea.
one thing i notice skimming the patch: snapshot.* is used to get the list of all snapshots -- didn't we removed similar wildcards from the other scripts because it barfs when the number of snapshots gets really big?
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Fix For: 1.3
>
> Attachments: solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-624) patch: Don't take snapshot if there are
no differences
Posted by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard "Trey" Hyde updated SOLR-624:
-------------------------------------
Attachment: SOLR-001.patch
Patch according to the Apache guidelines.
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Assignee: Grant Ingersoll
> Fix For: 1.3
>
> Attachments: SOLR-001.patch, solr.check.patch, solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-624) patch: Don't take snapshot if there
are no differences
Posted by "Bill Au (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612458#action_12612458 ]
Bill Au commented on SOLR-624:
------------------------------
Yes, the '*' will cost problem if there are many snapshots. The snappuller script has a shell command to determine the latest snapshot on the master. We can use the same command in snapshooter.
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Fix For: 1.3
>
> Attachments: solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-624) patch: Don't take snapshot if there
are no differences
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612237#action_12612237 ]
Shalin Shekhar Mangar commented on SOLR-624:
--------------------------------------------
Just curious, why would you cron snapshooter instead of using the postCommit/postOptimize hook?
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Fix For: 1.3
>
> Attachments: solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (SOLR-624) patch: Don't take snapshot if there are
no differences
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll reassigned SOLR-624:
------------------------------------
Assignee: Grant Ingersoll
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Assignee: Grant Ingersoll
> Fix For: 1.3
>
> Attachments: solr.check.patch, solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-624) patch: Don't take snapshot if there are
no differences
Posted by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard "Trey" Hyde updated SOLR-624:
-------------------------------------
Attachment: solr.check.patch
Updated patch resolves issues with wildcards with too many matches.
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Fix For: 1.3
>
> Attachments: solr.check.patch, solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-624) patch: Don't take snapshot if there
are no differences
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620045#action_12620045 ]
Grant Ingersoll commented on SOLR-624:
--------------------------------------
Richard,
Can you please generate the patch per the instructions at http://wiki.apache.org/solr/HowToContribute
Thanks,
Grant
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Assignee: Grant Ingersoll
> Fix For: 1.3
>
> Attachments: solr.check.patch, solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-624) patch: Don't take snapshot if there
are no differences
Posted by "Doug Steigerwald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612244#action_12612244 ]
Doug Steigerwald commented on SOLR-624:
---------------------------------------
If you have lots of cores committing very often (like we do), it can totally thrash your system. I was planning on implementing something like this at very soon. Glad someone else already did.
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Fix For: 1.3
>
> Attachments: solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SOLR-624) patch: Don't take snapshot if there are
no differences
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll resolved SOLR-624.
----------------------------------
Resolution: Fixed
Committed revision 683785.
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Assignee: Grant Ingersoll
> Fix For: 1.3
>
> Attachments: SOLR-001.patch, solr.check.patch, solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-624) patch: Don't take snapshot if there are
no differences
Posted by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard "Trey" Hyde updated SOLR-624:
-------------------------------------
Attachment: solr.check.patch
Patch to snapshooter
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Fix For: 1.3
>
> Attachments: solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-624) patch: Don't take snapshot if there
are no differences
Posted by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612262#action_12612262 ]
Richard "Trey" Hyde commented on SOLR-624:
------------------------------------------
The original patch was before postCommit/postOptimize hooks existed (or at least before I know about them) in the days before Solr was open sourced. Threw this quickly together today when I was annoyed when using snapshooter on the command line (testing MacOS, FreeBSD compatibility among other things).
It seems there is one other person that might find it useful.
> patch: Don't take snapshot if there are no differences
> ------------------------------------------------------
>
> Key: SOLR-624
> URL: https://issues.apache.org/jira/browse/SOLR-624
> Project: Solr
> Issue Type: Improvement
> Components: replication
> Affects Versions: 1.3
> Reporter: Richard "Trey" Hyde
> Fix For: 1.3
>
> Attachments: solr.check.patch
>
>
> This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.
> Adds a "check" argument to make sure there where changes to the index before taking the snap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.