You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Pushkar Raste (JIRA)" <ji...@apache.org> on 2016/06/13 15:47:20 UTC
[jira] [Updated] (SOLR-9207) PeerSync recovery failes if number of
updates requested is high
[ https://issues.apache.org/jira/browse/SOLR-9207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pushkar Raste updated SOLR-9207:
--------------------------------
Description:
{{PeerSync}} recovery fails if we request more than ~99K updates.
If update solrconfig to retain more {{tlogs}} to leverage https://issues.apache.org/jira/browse/SOLR-6359
During out testing we found out that recovery using {{PeerSync}} fails if we ask for more than ~99K updates, with following error
{code}
WARN PeerSync [RecoveryThread] - PeerSync: core=hold_shard1 url=<shardUrl>
exception talking to <leaderUrl>, failed
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got application/xml.
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="error"><str name="msg">application/x-www-form-urlencoded content length (4761994 bytes) exceeds upload limit of 2048 KB</str><in
t name="code">400</int></lst>
</response>
{code}
We arrived at ~99K with following match
* max_version_number = Long.MAX_VALUE = 9223372036854775807
* bytes per version number = 20 (on the wire as POST request sends version number as string)
* additional bytes for separator ,
* max_versions_in_single_request = 2MB/21 = ~99864
I could think of 2 ways to fix it
1. Ask for about updates in chunks of 90K inside {{PeerSync.requestUpdates()}}
2. Use application/octet-stream encoding
was:
{{PeerSync}} recovery fails if we request more than ~99K updates.
If update solrconfig to retain more {{tlogs}} to leverage https://issues.apache.org/jira/browse/SOLR-6359
During out testing we found out that recovery using {{PeerSync}} fails if we ask for more than ~99K updates, with following error
{code}
WARN PeerSync [RecoveryThread] - PeerSync: core=hold_shard1 url=<shardUrl>
exception talking to <leaderUrl>, failed
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got application/xml.
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="error"><str name="msg">application/x-www-form-urlencoded content length (4761994 bytes) exceeds upload limit of 2048 KB</str><in
t name="code">400</int></lst>
</response>
{code}
We arrived at ~99K with following match
* max_version_number = Long.MAX_VALUE = 9223372036854775807
* bytes per version number = 20 (on the wire as POST requestsends version number as string)
* additional bytes for separate ,
* max_versions_in_single_request = 2MB/21 = ~99864
I could think of 2 ways to fix it
1. Ask for about updates in chunks of 90K inside {{PeerSync.requestUpdates()}}
2. Use application/octet-stream encoding
> PeerSync recovery failes if number of updates requested is high
> ---------------------------------------------------------------
>
> Key: SOLR-9207
> URL: https://issues.apache.org/jira/browse/SOLR-9207
> Project: Solr
> Issue Type: Bug
> Affects Versions: 5.1, 6.0
> Reporter: Pushkar Raste
> Priority: Minor
>
> {{PeerSync}} recovery fails if we request more than ~99K updates.
> If update solrconfig to retain more {{tlogs}} to leverage https://issues.apache.org/jira/browse/SOLR-6359
> During out testing we found out that recovery using {{PeerSync}} fails if we ask for more than ~99K updates, with following error
> {code}
> WARN PeerSync [RecoveryThread] - PeerSync: core=hold_shard1 url=<shardUrl>
> exception talking to <leaderUrl>, failed
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got application/xml.
> <?xml version="1.0" encoding="UTF-8"?>
> <response>
> <lst name="error"><str name="msg">application/x-www-form-urlencoded content length (4761994 bytes) exceeds upload limit of 2048 KB</str><in
> t name="code">400</int></lst>
> </response>
> {code}
> We arrived at ~99K with following match
> * max_version_number = Long.MAX_VALUE = 9223372036854775807
> * bytes per version number = 20 (on the wire as POST request sends version number as string)
> * additional bytes for separator ,
> * max_versions_in_single_request = 2MB/21 = ~99864
> I could think of 2 ways to fix it
> 1. Ask for about updates in chunks of 90K inside {{PeerSync.requestUpdates()}}
> 2. Use application/octet-stream encoding
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org