[jira] [Commented] (SOLR-9091) Solr index restore silently copies the corrupt segments in the backup

["Hrishikesh Gadre (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276867#comment-15276867 ] 

Hrishikesh Gadre commented on SOLR-9091:
----------------------------------------

BTW Lucene uses CRC32 algorithm for computing checksum (LUCENE-2446). CRC32 is a just "error detection" code unlike MD5 (which produces one-way hash-codes). What it means that CRC32 checksums for two different files should not be compared for equality. (You can do that in case of MD5. e.g. during user registration - store the MD5 hash of the password in the database. When user attempts to login, you compute MD5 hash and check if it is "equal" to the one stored in the database). 

Here is a good discussion on this topic, http://www.dslreports.com/forum/r13525942-File-Integrity-Checksum-Algorithms-CRC32-vs-MD5

To summarize, I think we should NOT compare checksum of a file in the backup directory against the one in the current index directory. This checksum should ONLY be used to verify it the CRC32 hash of the file contents match the stored checksum (in the SAME file).

This problem will affect the "restore" operation as follows,

-> User backs up index files for Core A
-> User creates a new core (Core B) and index DIFFERENT documents.
-> Unfortunately the CRC32 code for an index file in core B match exactly with CRC32 code of a similarly named file in the BACKUP copy (Difficult but not impossible to happen).
-> User runs restore operation of core B using previously created BACKUP (Of Core A)
-> After restore user still sees SAME documents even though they were not part of original BACKUP.






 

> Solr index restore silently copies the corrupt segments in the backup
> ---------------------------------------------------------------------
>
>                 Key: SOLR-9091
>                 URL: https://issues.apache.org/jira/browse/SOLR-9091
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Hrishikesh Gadre
>
> The Solr core restore functionality uses following criteria to decide if a given file is copied from backup directory or from current index directory.
> case 1] File is available in both backup and current index directory
> --> Compare the checksum and file length
>   --> If checksum and length matching, copy the file from current working directory.
>  --> If the checksum and length doesn't match, copy the file from backup directory. 
> case 2] File is available in only in backup directory (This can happen for a newly created core without any data).
> --> Copy the file from backup directory. 
> Now the problem here is that we intentionally catch and ignore the error while reading the checksum for a file in the backup directory. Hence in case (2), it will result into restoration of a file without appropriate "checksum".
> Here is the relevant code snippet,
> https://github.com/apache/lucene-solr/blob/a5586d29b23f7d032e6d8f0cf8758e56b09e0208/solr/core/src/java/org/apache/solr/handler/RestoreCore.java#L82-L95



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org