You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Eungsop Yoo (JIRA)" <ji...@apache.org> on 2016/07/08 00:06:11 UTC
[jira] [Updated] (SOLR-9236) AutoAddReplicas feature with one
replica loses some documents not committed during failover
[ https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eungsop Yoo updated SOLR-9236:
------------------------------
Description:
I need to index huge amount of logs, so I decide to use AutoAddReplica feature with only one replica.
When using AutoAddReplicas with one replica, some benefits are expected.
- no redundant data files for replicas
-- saving disk usage
- best indexing performance
I expected that Solr fails over just like HBase.
The feature worked almost as it was expected, except for some missing documents during failover.
I found two reasons for the missing.
1. The leader replica does not replay any transaction logs. But when there is only one replica, it should be the leader.
So I made the leader replica replay the transaction logs when using AutoAddReplicas with on replica.
But the above fix did not resolve the problem.
2. As failover occurred, the transaction log directory had a deeper directory depth. Just like this, tlog/tlog/tlog/...
The transaction log could not be replayed, because the transaction log directory was changed during failover.
So I made the transaction log directory not changed during failover.
After these fixes, AutoAddReplicas with one replica fails over well without losing any documents.
was:
I need to index huge amount of logs, so I decide to use AutoAddReplica feature with only one replica.
When using AutoAddReplicas with one replica, some benefits are expected.
- no redundant data files for replicas
-- saving disk usage
- best indexing performance
I expected that Solr fails over just like HBase.
The feature worked almost as it was expected, except for some missing documents during failover.
I found two regions for the missing.
1. The leader replica does not replay any transaction logs. But when there is only one replica, it should be the leader.
So I made the leader replica replay the transaction logs when using AutoAddReplicas with on replica.
But the above fix did not resolve the problem.
2. As failover occurred, the transaction log directory had a deeper directory depth. Just like this, tlog/tlog/tlog/...
The transaction log could not be replayed, because the transaction log directory was changed during failover.
So I made the transaction log directory not changed during failover.
After these fixes, AutoAddReplicas with one replica fails over well without losing any documents.
> AutoAddReplicas feature with one replica loses some documents not committed during failover
> -------------------------------------------------------------------------------------------
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
> Issue Type: Bug
> Components: hdfs, SolrCloud
> Reporter: Eungsop Yoo
> Assignee: Mark Miller
> Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing documents during failover.
> I found two reasons for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log directory was changed during failover.
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without losing any documents.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org