You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Stephen Lewis <ia...@stephen-lewis.net> on 2016/08/23 02:41:19 UTC

Error upgrading from 6.0 to 6.1

Hello,

I have a question about updating a solr cloud cluster servers in place. I
have a scripted method for updating a solr cloud in place, which works
consistently to up/down grade between 6.0.0 and 6.0.1 (in our test
environment), but hits an error consistently when going from either to solr
6.1.0. Each server is hosting a single solr node, and each shard has a
replication factor of 3.

​The way the script works is as follows. For each instance:

1. Pull the instance from serving requests and drain.
2. Delete the replica from the collection (but leave the index and data)
3. If that node was the leader, force a leader election (solr is not
accepting writes at this time, so this is safe)
4. Run a bootstrapping script on the remote machine (which installs a
particular solr version, and is otherwise idempotent)
5. Once the instance is updated and solr is confirmed, add the node as a
replica where it used to be
6. Wait for recovery
7. Reserve requests from this node.

​As mentioned, this hasn't shown any problem switching between versions
6.0.0 and 6.0.1, but when I try to use this to upgrade to solr 6.1.0, the
"ADDREPLICA" command fails as follows:

"status":500,"QTime":65},"failure":{"172.18.6.68:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
from server at http://172.18.6.68:8983/solr: Expected mime type
application/octet-stream but got text/html

I've included​ the full log below as an attachment. The exact request being
served is the following:

http://52.91.138.30:8983/solr/admin/collections?wt=json&action=ADDREPLICA&collection=panopto&shard=shard2&node=172.18.6.68:8983_solr

I didn't see any special actions which needed to be taken when upgrading to
6.1.0. Is there perhaps something wrong in my upgrade methodology or
anything else you're aware of which may be related?

Thanks for your help!
Stephen

-- 
www.stephen-lewis.net

Re: Error upgrading from 6.0 to 6.1

Posted by Stephen Lewis <sl...@panopto.com>.
Erick, Shawn, you're right on both counts. Mixed jar versions are happening
in both cases, and only lead to a fatal error on on the upgrade to 6.1.0.
So there was a big gap in my upgrading methodology. I've confirmed that
fixing the bootstrapping script allows the upgrade and that the correct jar
files are being loaded after the fix.

Sorry for the confusion with the attachment. I've included the log below
just in case it is helpful to anyone listening.

Thanks again!

Best,
Stephen

{msg=org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/
Map;,trace=java.lang.NoSuchMethodError: org.apache.solr.common.cloud.
ZkStateReader.getClusterProps()Ljava/util/Map;
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:743)
at org.apache.solr.handler.admin.CoreAdminOperation$1.call(
CoreAdminOperation.java:134)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.
call(CoreAdminHandler.java:351)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(
CoreAdminHandler.java:153)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(
RequestHandlerBase.java:155)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
HttpSolrCall.java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDispatchFilter.java:229)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDispatchFilter.java:184)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(
ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(
SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.
doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.
doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.
doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.
doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.
handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(
HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
,code=500}

On Tue, Aug 23, 2016 at 6:00 AM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 8/22/2016 9:18 PM, Stephen Lewis wrote:
> > Oops, apologies for my confusing grammar and for missing the
> > attachment. The intro sentence should have read "I have a question
> > about upgrading a solr cloud cluster in place." I've actually attached
> > the log below this time.
>
> The mailing list eats most attachments.  Sometimes they make it through,
> usually they don't, and I never can see what causes the difference.
> Your attachment did not make it through.
>
> For us to see it, you will need to to place the log somewhere on the
> Internet and provide a URL to access it.
>
> When you get a message saying that application/octet-stream was expected
> but text/html is received instead, it usually means what was received
> from the remote server was an error page, instead of the javabin
> response that was expected.  To see what that error is, you'll need to
> check the log on the remote server -- in this case, the server with IP
> address 172.18.6.68.
>
> Further down in the thread you did mention a NoSuchMethodError.  If
> that's the error message from 172.18.6.68, then I agree with Erick's
> assessment.  You've probably got multiple versions of Solr jars on your
> classpath.
>
> Best guess is that your bootstrapping step copies the install directory
> without deleting anything from the target, which would *add* jars to
> server/solr-webapp/webapp/WEB-INF/lib.  The jars in the two versions of
> Solr do not have the same names -- the full version number is part of
> the filename.
>
> I can anticipate a possible next question:  Why did it work when
> upgrading from 6.0.0 to 6.0.1?  The answer:  Mixing jars would most
> likely work with those two versions, because that's a bugfix release,
> and bugfix releases are usually 100 percent API/ABI compatible with the
> previous version.  Breaks in that compatibility are expected in minor
> version upgrades on the server side, especially in code relating to
> SolrCloud.  That code is evolving *VERY* rapidly.
>
> If you do not see evidence supporting the multiple-jar-version idea,
> then you may need to provide access to the logfile.
>
> We do *try* to ensure API/ABI compatibility on the the *client* side so
> user programs can update SolrJ to a new minor version without
> recompiling ... but even that is not guaranteed.
>
> Thanks,
> Shawn
>
>


-- 
Stephen

(206)753-9320
stephen-lewis.net

Re: Error upgrading from 6.0 to 6.1

Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/22/2016 9:18 PM, Stephen Lewis wrote:
> Oops, apologies for my confusing grammar and for missing the
> attachment. The intro sentence should have read "I have a question
> about upgrading a solr cloud cluster in place." I've actually attached
> the log below this time.

The mailing list eats most attachments.  Sometimes they make it through,
usually they don't, and I never can see what causes the difference. 
Your attachment did not make it through.

For us to see it, you will need to to place the log somewhere on the
Internet and provide a URL to access it.

When you get a message saying that application/octet-stream was expected
but text/html is received instead, it usually means what was received
from the remote server was an error page, instead of the javabin
response that was expected.  To see what that error is, you'll need to
check the log on the remote server -- in this case, the server with IP
address 172.18.6.68.

Further down in the thread you did mention a NoSuchMethodError.  If
that's the error message from 172.18.6.68, then I agree with Erick's
assessment.  You've probably got multiple versions of Solr jars on your
classpath.

Best guess is that your bootstrapping step copies the install directory
without deleting anything from the target, which would *add* jars to
server/solr-webapp/webapp/WEB-INF/lib.  The jars in the two versions of
Solr do not have the same names -- the full version number is part of
the filename.

I can anticipate a possible next question:  Why did it work when
upgrading from 6.0.0 to 6.0.1?  The answer:  Mixing jars would most
likely work with those two versions, because that's a bugfix release,
and bugfix releases are usually 100 percent API/ABI compatible with the
previous version.  Breaks in that compatibility are expected in minor
version upgrades on the server side, especially in code relating to
SolrCloud.  That code is evolving *VERY* rapidly.

If you do not see evidence supporting the multiple-jar-version idea,
then you may need to provide access to the logfile.

We do *try* to ensure API/ABI compatibility on the the *client* side so
user programs can update SolrJ to a new minor version without
recompiling ... but even that is not guaranteed.

Thanks,
Shawn

Re: Error upgrading from 6.0 to 6.1

Posted by Erick Erickson <er...@gmail.com>.
That's usually an indication that your classpath has old and new jars
in it. When you start Solr,
the directories from which all the jars are listed in the log file. My
guess is that if you examine them
you'll see jar files loaded from both 6.0 and some from 6.1 and you
need to figure out how
that happened and undo it....

Best,
Erick

On Mon, Aug 22, 2016 at 8:31 PM, Stephen Lewis <ia...@stephen-lewis.net> wrote:
> In particular, I see this line. Was there perhaps a deprecation of a method
> or something changed about cluster properties?
>
> <head>
> <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\"/>
> <title>Error 500
> {msg=org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/Map;,trace=java.lang.NoSuchMethodError:
> org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/Map;
>
> On Mon, Aug 22, 2016 at 8:18 PM, Stephen Lewis <ia...@stephen-lewis.net>
> wrote:
>
>> Oops, apologies for my confusing grammar and for missing the attachment.
>> The intro sentence should have read "I have a question about upgrading a
>> solr cloud cluster in place." I've actually attached the log below this
>> time.
>>
>> Thanks again,
>> Stephen
>>
>> On Mon, Aug 22, 2016 at 7:41 PM, Stephen Lewis <ia...@stephen-lewis.net>
>> wrote:
>>
>>> Hello,
>>>
>>> I have a question about updating a solr cloud cluster servers in place. I
>>> have a scripted method for updating a solr cloud in place, which works
>>> consistently to up/down grade between 6.0.0 and 6.0.1 (in our test
>>> environment), but hits an error consistently when going from either to solr
>>> 6.1.0. Each server is hosting a single solr node, and each shard has a
>>> replication factor of 3.
>>>
>>> The way the script works is as follows. For each instance:
>>>
>>> 1. Pull the instance from serving requests and drain.
>>> 2. Delete the replica from the collection (but leave the index and data)
>>> 3. If that node was the leader, force a leader election (solr is not
>>> accepting writes at this time, so this is safe)
>>> 4. Run a bootstrapping script on the remote machine (which installs a
>>> particular solr version, and is otherwise idempotent)
>>> 5. Once the instance is updated and solr is confirmed, add the node as a
>>> replica where it used to be
>>> 6. Wait for recovery
>>> 7. Reserve requests from this node.
>>>
>>> As mentioned, this hasn't shown any problem switching between versions
>>> 6.0.0 and 6.0.1, but when I try to use this to upgrade to solr 6.1.0, the
>>> "ADDREPLICA" command fails as follows:
>>>
>>> "status":500,"QTime":65},"failure":{"172.18.6.68:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://172.18.6.68:8983/solr: Expected mime type application/octet-stream but got text/html
>>>
>>> I've included the full log below as an attachment. The exact request
>>> being served is the following:
>>>
>>> http://52.91.138.30:8983/solr/admin/collections?wt=json&acti
>>> on=ADDREPLICA&collection=panopto&shard=shard2&node=172.18.6.68:8983_solr
>>>
>>> I didn't see any special actions which needed to be taken when upgrading
>>> to 6.1.0. Is there perhaps something wrong in my upgrade methodology or
>>> anything else you're aware of which may be related?
>>>
>>> Thanks for your help!
>>> Stephen
>>>
>>> --
>>> www.stephen-lewis.net
>>>
>>
>>
>>
>> --
>> www.stephen-lewis.net
>>
>
>
>
> --
> www.stephen-lewis.net

Re: Error upgrading from 6.0 to 6.1

Posted by Stephen Lewis <ia...@stephen-lewis.net>.
In particular, I see this line. Was there perhaps a deprecation of a method
or something changed about cluster properties?

<head>
<meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\"/>
<title>Error 500
{msg=org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/Map;,trace=java.lang.NoSuchMethodError:
org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/Map;

On Mon, Aug 22, 2016 at 8:18 PM, Stephen Lewis <ia...@stephen-lewis.net>
wrote:

> Oops, apologies for my confusing grammar and for missing the attachment.
> The intro sentence should have read "I have a question about upgrading a
> solr cloud cluster in place." I've actually attached the log below this
> time.
>
> Thanks again,
> Stephen
>
> On Mon, Aug 22, 2016 at 7:41 PM, Stephen Lewis <ia...@stephen-lewis.net>
> wrote:
>
>> Hello,
>>
>> I have a question about updating a solr cloud cluster servers in place. I
>> have a scripted method for updating a solr cloud in place, which works
>> consistently to up/down grade between 6.0.0 and 6.0.1 (in our test
>> environment), but hits an error consistently when going from either to solr
>> 6.1.0. Each server is hosting a single solr node, and each shard has a
>> replication factor of 3.
>>
>> ​The way the script works is as follows. For each instance:
>>
>> 1. Pull the instance from serving requests and drain.
>> 2. Delete the replica from the collection (but leave the index and data)
>> 3. If that node was the leader, force a leader election (solr is not
>> accepting writes at this time, so this is safe)
>> 4. Run a bootstrapping script on the remote machine (which installs a
>> particular solr version, and is otherwise idempotent)
>> 5. Once the instance is updated and solr is confirmed, add the node as a
>> replica where it used to be
>> 6. Wait for recovery
>> 7. Reserve requests from this node.
>>
>> ​As mentioned, this hasn't shown any problem switching between versions
>> 6.0.0 and 6.0.1, but when I try to use this to upgrade to solr 6.1.0, the
>> "ADDREPLICA" command fails as follows:
>>
>> "status":500,"QTime":65},"failure":{"172.18.6.68:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://172.18.6.68:8983/solr: Expected mime type application/octet-stream but got text/html
>>
>> I've included​ the full log below as an attachment. The exact request
>> being served is the following:
>>
>> http://52.91.138.30:8983/solr/admin/collections?wt=json&acti
>> on=ADDREPLICA&collection=panopto&shard=shard2&node=172.18.6.68:8983_solr
>>
>> I didn't see any special actions which needed to be taken when upgrading
>> to 6.1.0. Is there perhaps something wrong in my upgrade methodology or
>> anything else you're aware of which may be related?
>>
>> Thanks for your help!
>> Stephen
>>
>> --
>> www.stephen-lewis.net
>>
>
>
>
> --
> www.stephen-lewis.net
>



-- 
www.stephen-lewis.net

Re: Error upgrading from 6.0 to 6.1

Posted by Stephen Lewis <ia...@stephen-lewis.net>.
Oops, apologies for my confusing grammar and for missing the attachment.
The intro sentence should have read "I have a question about upgrading a
solr cloud cluster in place." I've actually attached the log below this
time.

Thanks again,
Stephen

On Mon, Aug 22, 2016 at 7:41 PM, Stephen Lewis <ia...@stephen-lewis.net>
wrote:

> Hello,
>
> I have a question about updating a solr cloud cluster servers in place. I
> have a scripted method for updating a solr cloud in place, which works
> consistently to up/down grade between 6.0.0 and 6.0.1 (in our test
> environment), but hits an error consistently when going from either to solr
> 6.1.0. Each server is hosting a single solr node, and each shard has a
> replication factor of 3.
>
> ​The way the script works is as follows. For each instance:
>
> 1. Pull the instance from serving requests and drain.
> 2. Delete the replica from the collection (but leave the index and data)
> 3. If that node was the leader, force a leader election (solr is not
> accepting writes at this time, so this is safe)
> 4. Run a bootstrapping script on the remote machine (which installs a
> particular solr version, and is otherwise idempotent)
> 5. Once the instance is updated and solr is confirmed, add the node as a
> replica where it used to be
> 6. Wait for recovery
> 7. Reserve requests from this node.
>
> ​As mentioned, this hasn't shown any problem switching between versions
> 6.0.0 and 6.0.1, but when I try to use this to upgrade to solr 6.1.0, the
> "ADDREPLICA" command fails as follows:
>
> "status":500,"QTime":65},"failure":{"172.18.6.68:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://172.18.6.68:8983/solr: Expected mime type application/octet-stream but got text/html
>
> I've included​ the full log below as an attachment. The exact request
> being served is the following:
>
> http://52.91.138.30:8983/solr/admin/collections?wt=json&acti
> on=ADDREPLICA&collection=panopto&shard=shard2&node=172.18.6.68:8983_solr
>
> I didn't see any special actions which needed to be taken when upgrading
> to 6.1.0. Is there perhaps something wrong in my upgrade methodology or
> anything else you're aware of which may be related?
>
> Thanks for your help!
> Stephen
>
> --
> www.stephen-lewis.net
>



-- 
www.stephen-lewis.net