You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jayendra Patil (JIRA)" <ji...@apache.org> on 2010/10/15 02:33:32 UTC
[jira] Created: (SOLR-2156) Solr Replication - SnapPuller fails to
clean Old Index Directories on Full Copy
Solr Replication - SnapPuller fails to clean Old Index Directories on Full Copy
-------------------------------------------------------------------------------
Key: SOLR-2156
URL: https://issues.apache.org/jira/browse/SOLR-2156
Project: Solr
Issue Type: Improvement
Components: replication (java)
Affects Versions: 4.0
Reporter: Jayendra Patil
We are working on the Solr trunk ....and have a Master and Two slaves configuration .....
Our indexing consists of Periodic Full and Incremental index building on the master and replication on the slaves.
When a Full indexing (clean and rebuild) is performed, we always end with an extra index folder copy, which holds the complete index and hence the size just grows on, on the slaves.
e.g.
drwxr-xr-x 2 tomcat tomcat 4096 2010-10-09 12:10 index
drwxr-xr-x 2 tomcat tomcat 4096 2010-10-11 09:43 index.20101009120649
drwxr-xr-x 2 tomcat tomcat 4096 2010-10-12 10:27 index.20101011094043
-rw-r--r-- 1 tomcat tomcat 75 2010-10-11 09:43 index.properties
-rw-r--r-- 1 tomcat tomcat 422 2010-10-12 10:26 replication.properties
drwxr-xr-x 2 tomcat tomcat 68 2010-10-12 10:27 spellchecker
Where index.20101011094043 is the active index and the other index.xxx directories are no more used.
The SnapPuller deletes the temporary Index directory, but does not delete the old one when the switch is performed for the full copy.
The below code should do the trick.
boolean fetchLatestIndex(SolrCore core) throws IOException {
..........
} finally {
if(deleteTmpIdxDir) {
delTree(tmpIndexDir);
} else {
// Delete the old index directory, as the flag is set only after the full copy is performed
delTree(indexDir);
}
}
.........
}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2156) Solr Replication - SnapPuller fails
to clean Old Index Directories on Full Copy
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928601#action_12928601 ]
Noble Paul commented on SOLR-2156:
----------------------------------
The patch looks good..
> Solr Replication - SnapPuller fails to clean Old Index Directories on Full Copy
> -------------------------------------------------------------------------------
>
> Key: SOLR-2156
> URL: https://issues.apache.org/jira/browse/SOLR-2156
> Project: Solr
> Issue Type: Improvement
> Components: replication (java)
> Affects Versions: 4.0
> Reporter: Jayendra Patil
> Attachments: Solr-2156_SnapPuller.patch
>
>
> We are working on the Solr trunk ....and have a Master and Two slaves configuration .....
> Our indexing consists of Periodic Full and Incremental index building on the master and replication on the slaves.
> When a Full indexing (clean and rebuild) is performed, we always end with an extra index folder copy, which holds the complete index and hence the size just grows on, on the slaves.
> e.g.
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-09 12:10 index
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-11 09:43 index.20101009120649
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-12 10:27 index.20101011094043
> -rw-r--r-- 1 tomcat tomcat 75 2010-10-11 09:43 index.properties
> -rw-r--r-- 1 tomcat tomcat 422 2010-10-12 10:26 replication.properties
> drwxr-xr-x 2 tomcat tomcat 68 2010-10-12 10:27 spellchecker
> Where index.20101011094043 is the active index and the other index.xxx directories are no more used.
> The SnapPuller deletes the temporary Index directory, but does not delete the old one when the switch is performed for the full copy.
> The below code should do the trick.
> boolean fetchLatestIndex(SolrCore core) throws IOException {
> ..........
> } finally {
> if(deleteTmpIdxDir) {
> delTree(tmpIndexDir);
> } else {
> // Delete the old index directory, as the flag is set only after the full copy is performed
> delTree(indexDir);
> }
> }
> .........
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (SOLR-2156) Solr Replication - SnapPuller fails to
clean Old Index Directories on Full Copy
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lance Norskog updated SOLR-2156:
--------------------------------
Comment: was deleted
(was: When does 'deleteTmpIdxDir' ever get set false? When can this change ever get run?)
> Solr Replication - SnapPuller fails to clean Old Index Directories on Full Copy
> -------------------------------------------------------------------------------
>
> Key: SOLR-2156
> URL: https://issues.apache.org/jira/browse/SOLR-2156
> Project: Solr
> Issue Type: Improvement
> Components: replication (java)
> Affects Versions: 4.0
> Reporter: Jayendra Patil
> Attachments: Solr-2156_SnapPuller.patch
>
>
> We are working on the Solr trunk ....and have a Master and Two slaves configuration .....
> Our indexing consists of Periodic Full and Incremental index building on the master and replication on the slaves.
> When a Full indexing (clean and rebuild) is performed, we always end with an extra index folder copy, which holds the complete index and hence the size just grows on, on the slaves.
> e.g.
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-09 12:10 index
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-11 09:43 index.20101009120649
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-12 10:27 index.20101011094043
> -rw-r--r-- 1 tomcat tomcat 75 2010-10-11 09:43 index.properties
> -rw-r--r-- 1 tomcat tomcat 422 2010-10-12 10:26 replication.properties
> drwxr-xr-x 2 tomcat tomcat 68 2010-10-12 10:27 spellchecker
> Where index.20101011094043 is the active index and the other index.xxx directories are no more used.
> The SnapPuller deletes the temporary Index directory, but does not delete the old one when the switch is performed for the full copy.
> The below code should do the trick.
> boolean fetchLatestIndex(SolrCore core) throws IOException {
> ..........
> } finally {
> if(deleteTmpIdxDir) {
> delTree(tmpIndexDir);
> } else {
> // Delete the old index directory, as the flag is set only after the full copy is performed
> delTree(indexDir);
> }
> }
> .........
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2156) Solr Replication - SnapPuller fails
to clean Old Index Directories on Full Copy
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928926#action_12928926 ]
Lance Norskog commented on SOLR-2156:
-------------------------------------
When does 'deleteTmpIdxDir' ever get set false? When can this change ever get run?
> Solr Replication - SnapPuller fails to clean Old Index Directories on Full Copy
> -------------------------------------------------------------------------------
>
> Key: SOLR-2156
> URL: https://issues.apache.org/jira/browse/SOLR-2156
> Project: Solr
> Issue Type: Improvement
> Components: replication (java)
> Affects Versions: 4.0
> Reporter: Jayendra Patil
> Attachments: Solr-2156_SnapPuller.patch
>
>
> We are working on the Solr trunk ....and have a Master and Two slaves configuration .....
> Our indexing consists of Periodic Full and Incremental index building on the master and replication on the slaves.
> When a Full indexing (clean and rebuild) is performed, we always end with an extra index folder copy, which holds the complete index and hence the size just grows on, on the slaves.
> e.g.
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-09 12:10 index
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-11 09:43 index.20101009120649
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-12 10:27 index.20101011094043
> -rw-r--r-- 1 tomcat tomcat 75 2010-10-11 09:43 index.properties
> -rw-r--r-- 1 tomcat tomcat 422 2010-10-12 10:26 replication.properties
> drwxr-xr-x 2 tomcat tomcat 68 2010-10-12 10:27 spellchecker
> Where index.20101011094043 is the active index and the other index.xxx directories are no more used.
> The SnapPuller deletes the temporary Index directory, but does not delete the old one when the switch is performed for the full copy.
> The below code should do the trick.
> boolean fetchLatestIndex(SolrCore core) throws IOException {
> ..........
> } finally {
> if(deleteTmpIdxDir) {
> delTree(tmpIndexDir);
> } else {
> // Delete the old index directory, as the flag is set only after the full copy is performed
> delTree(indexDir);
> }
> }
> .........
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2156) Solr Replication - SnapPuller fails
to clean Old Index Directories on Full Copy
Posted by "Jayendra Patil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928928#action_12928928 ]
Jayendra Patil commented on SOLR-2156:
--------------------------------------
There are different conditions in which the flag gets set to false.
If the Index is stale (Common Index files of slave does not exist in master
or are of different size, or with different timestamp - isIndexStale method)
or more newer than the master (slave generation > master generation) a full
index download is performed.
In our case we do a clean build so the files on on slave don't exist on
master and hence the flag is set to false.
> Solr Replication - SnapPuller fails to clean Old Index Directories on Full Copy
> -------------------------------------------------------------------------------
>
> Key: SOLR-2156
> URL: https://issues.apache.org/jira/browse/SOLR-2156
> Project: Solr
> Issue Type: Improvement
> Components: replication (java)
> Affects Versions: 4.0
> Reporter: Jayendra Patil
> Attachments: Solr-2156_SnapPuller.patch
>
>
> We are working on the Solr trunk ....and have a Master and Two slaves configuration .....
> Our indexing consists of Periodic Full and Incremental index building on the master and replication on the slaves.
> When a Full indexing (clean and rebuild) is performed, we always end with an extra index folder copy, which holds the complete index and hence the size just grows on, on the slaves.
> e.g.
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-09 12:10 index
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-11 09:43 index.20101009120649
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-12 10:27 index.20101011094043
> -rw-r--r-- 1 tomcat tomcat 75 2010-10-11 09:43 index.properties
> -rw-r--r-- 1 tomcat tomcat 422 2010-10-12 10:26 replication.properties
> drwxr-xr-x 2 tomcat tomcat 68 2010-10-12 10:27 spellchecker
> Where index.20101011094043 is the active index and the other index.xxx directories are no more used.
> The SnapPuller deletes the temporary Index directory, but does not delete the old one when the switch is performed for the full copy.
> The below code should do the trick.
> boolean fetchLatestIndex(SolrCore core) throws IOException {
> ..........
> } finally {
> if(deleteTmpIdxDir) {
> delTree(tmpIndexDir);
> } else {
> // Delete the old index directory, as the flag is set only after the full copy is performed
> delTree(indexDir);
> }
> }
> .........
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (SOLR-2156) Solr Replication - SnapPuller fails to
clean Old Index Directories on Full Copy
Posted by "Jayendra Patil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jayendra Patil updated SOLR-2156:
---------------------------------
Attachment: Solr-2156_SnapPuller.patch
Attached the Fix.
> Solr Replication - SnapPuller fails to clean Old Index Directories on Full Copy
> -------------------------------------------------------------------------------
>
> Key: SOLR-2156
> URL: https://issues.apache.org/jira/browse/SOLR-2156
> Project: Solr
> Issue Type: Improvement
> Components: replication (java)
> Affects Versions: 4.0
> Reporter: Jayendra Patil
> Attachments: Solr-2156_SnapPuller.patch
>
>
> We are working on the Solr trunk ....and have a Master and Two slaves configuration .....
> Our indexing consists of Periodic Full and Incremental index building on the master and replication on the slaves.
> When a Full indexing (clean and rebuild) is performed, we always end with an extra index folder copy, which holds the complete index and hence the size just grows on, on the slaves.
> e.g.
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-09 12:10 index
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-11 09:43 index.20101009120649
> drwxr-xr-x 2 tomcat tomcat 4096 2010-10-12 10:27 index.20101011094043
> -rw-r--r-- 1 tomcat tomcat 75 2010-10-11 09:43 index.properties
> -rw-r--r-- 1 tomcat tomcat 422 2010-10-12 10:26 replication.properties
> drwxr-xr-x 2 tomcat tomcat 68 2010-10-12 10:27 spellchecker
> Where index.20101011094043 is the active index and the other index.xxx directories are no more used.
> The SnapPuller deletes the temporary Index directory, but does not delete the old one when the switch is performed for the full copy.
> The below code should do the trick.
> boolean fetchLatestIndex(SolrCore core) throws IOException {
> ..........
> } finally {
> if(deleteTmpIdxDir) {
> delTree(tmpIndexDir);
> } else {
> // Delete the old index directory, as the flag is set only after the full copy is performed
> delTree(indexDir);
> }
> }
> .........
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org