[jira] [Commented] (SOLR-9207) PeerSync recovery failes if number of updates requested is high

["Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-9207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361557#comment-15361557 ] 

Shalin Shekhar Mangar commented on SOLR-9207:
---------------------------------------------

Thanks Pushkar! The patch looks good to me except that the testing is not adequate. The one test modified by the patch is RecoveryAfterSoftCommitTest which is designed to never trigger PeerSync at all. I think we should enable useRangeVersions by default and randomly set it to false during tests for good coverage.

> PeerSync recovery failes if number of updates requested is high
> ---------------------------------------------------------------
>
>                 Key: SOLR-9207
>                 URL: https://issues.apache.org/jira/browse/SOLR-9207
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 5.1, 6.0
>            Reporter: Pushkar Raste
>            Assignee: Shalin Shekhar Mangar
>            Priority: Minor
>         Attachments: SOLR-9207.patch
>
>
> {{PeerSync}} recovery fails if we request more than ~99K updates. 
> If update solrconfig to retain more {{tlogs}} to leverage https://issues.apache.org/jira/browse/SOLR-6359
> During out testing we found out that recovery using {{PeerSync}} fails if we ask for more than ~99K updates, with following error
> {code}
>  WARN  PeerSync [RecoveryThread] - PeerSync: core=hold_shard1 url=<shardUrl>
> exception talking to <leaderUrl>, failed
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got application/xml. 
> <?xml version="1.0" encoding="UTF-8"?>
> <response>
> <lst name="error"><str name="msg">application/x-www-form-urlencoded content length (4761994 bytes) exceeds upload limit of 2048 KB</str><in
> t name="code">400</int></lst>
> </response>
> {code}
> We arrived at ~99K with following match
> * max_version_number = Long.MAX_VALUE = 9223372036854775807  
> * bytes per version number =  20 (on the wire as POST request sends version number as string)
> * additional bytes for separator ,
> * max_versions_in_single_request = 2MB/21 = ~99864
> I could think of 2 ways to fix it
> 1. Ask for about updates in chunks of 90K inside {{PeerSync.requestUpdates()}}
> 2. Use application/octet-stream encoding 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org