You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Hoss Man (JIRA)" <ji...@apache.org> on 2012/12/03 00:05:58 UTC
[jira] [Created] (SOLR-4136) SolrCloud bugs when servlet context
contains "/" or "_"
Hoss Man created SOLR-4136:
------------------------------
Summary: SolrCloud bugs when servlet context contains "/" or "_"
Key: SOLR-4136
URL: https://issues.apache.org/jira/browse/SOLR-4136
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 4.0
Reporter: Hoss Man
Assignee: Hoss Man
Attachments: SOLR-4136.patch
SolrCloud does not work properly with non-trivial values for "hostContext" (ie: the servlet context path). In particular...
* Using a hostContext containing a "/" (ie: a servlet context with a subdir path, semi-common among people who organize webapps hierarchically for lod blanacer rules) is explicitly forbidden in ZkController because of how the hostContext is used to build a ZK nodeName
* Using a hostContext containing a "_" causes problems in OverseerCollectionProcessor where it assumes all "_" characters should be converted to "/" to reconstitute a URL from nodeName (NOTE: this code specifically has a TODO to fix this, and then has a subsequent TODO about assuming "http://" labeled "this sucks")
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4136) SolrCloud bugs when servlet context
contains "/" or "_"
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510710#comment-13510710 ]
Mark Miller commented on SOLR-4136:
-----------------------------------
Yeah, I've looked at this in the past because we have problems with _ in the url's the way things are now (there are comments to that affect). Never got to adding the right info to ZK yet though. I don't remember thinking of a silver bullet that seemed like the clear way to go.
Take a stab, I'll try and catch up soon.
> SolrCloud bugs when servlet context contains "/" or "_"
> -------------------------------------------------------
>
> Key: SOLR-4136
> URL: https://issues.apache.org/jira/browse/SOLR-4136
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.0
> Reporter: Hoss Man
> Assignee: Hoss Man
> Attachments: SOLR-4136.patch
>
>
> SolrCloud does not work properly with non-trivial values for "hostContext" (ie: the servlet context path). In particular...
> * Using a hostContext containing a "/" (ie: a servlet context with a subdir path, semi-common among people who organize webapps hierarchically for lod blanacer rules) is explicitly forbidden in ZkController because of how the hostContext is used to build a ZK nodeName
> * Using a hostContext containing a "\_" causes problems in OverseerCollectionProcessor where it assumes all "\_" characters should be converted to "/" to reconstitute a URL from nodeName (NOTE: this code specifically has a TODO to fix this, and then has a subsequent TODO about assuming "http://" labeled "this sucks")
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4136) SolrCloud bugs when servlet context
contains "/" or "_"
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510698#comment-13510698 ]
Hoss Man commented on SOLR-4136:
--------------------------------
Been poking around the SolrCloud/zk code ... fun times.
>From what i can tell, we don't record anywhere in zookeeper the mapping of "nodeName" -> "baseURL" for the various solr nodes in a solr cloud cluster. We _do_ seem evidently record the baseUrl associated with a nodeName in the info about each _replica_ -- but that information is per collection & shard, so as is it doesn't really help in the general case of the bad code in OverseerCollectionProcessor.
Three options occur to me...
1) We could consider adding these mappins to ZK as 1st order info. possibly by adding some data to the ephemeral "liveNodes" path for each node, so code like OverseerCollectionProcessor could just ask for the data of each liveNode to know it's baseUrl ... but i'm not sure how far down that rabithole we want to go (i don't really know the performance characteristics of ZK enough to know if it's a good idea to have code doing lots of those kinds of lookups ad-hoc)
2) we could cheat: we could add something like this to ClusterState...
{code}
private final Map<String,String> baseUrls;
public String getBaseUrl(final String nodeName);
{code}
...and populate the baseUrls Map in the constructor based on the properties found when looping over every collections->slice->replica. The only question is what to do if/when two diff collections/slice/replica in the clusterstate disagree about the baseUrl? (assertion failed?)
3) We could improve the kludge to be a bit less kludgy: OverseerCollectionProcessor (and possibly other places) currently assume that a baseUrl can be computed from a nodeName by replacing all "\_" with "/" -- if we change that substitution to only apply to the _first_ "\_" in the nodeName, and combine it with some URL decoding on the "hostContext" portion of the nodeName (to match my suggested improvement in theprevious patch) i think we would have a fairly safe way of bi-directinally converting nodeName<->URL regardless of what's in the hostContext - because hostnames and ports can't ever have "\_" in them. (this wouldn't address the "http://" kludge, but that assumption seems to be more pervasive - we can fight that battle another day)
--
Option #3 seems the invasive for now, so unless mark/yonik/sami/somebody chimes in with more encouragment to go down one of the other routes, i'll take a stab at #3 and see what other problems i encounter.
> SolrCloud bugs when servlet context contains "/" or "_"
> -------------------------------------------------------
>
> Key: SOLR-4136
> URL: https://issues.apache.org/jira/browse/SOLR-4136
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.0
> Reporter: Hoss Man
> Assignee: Hoss Man
> Attachments: SOLR-4136.patch
>
>
> SolrCloud does not work properly with non-trivial values for "hostContext" (ie: the servlet context path). In particular...
> * Using a hostContext containing a "/" (ie: a servlet context with a subdir path, semi-common among people who organize webapps hierarchically for lod blanacer rules) is explicitly forbidden in ZkController because of how the hostContext is used to build a ZK nodeName
> * Using a hostContext containing a "\_" causes problems in OverseerCollectionProcessor where it assumes all "\_" characters should be converted to "/" to reconstitute a URL from nodeName (NOTE: this code specifically has a TODO to fix this, and then has a subsequent TODO about assuming "http://" labeled "this sucks")
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-4136) SolrCloud bugs when servlet context
contains "/" or "_"
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated SOLR-4136:
---------------------------
Attachment: SOLR-4136.patch
Context...
* http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Calpine.DEB.2.02.1211292004430.2543@frisbee%3E
* http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3C551C5E62-0520-42A2-BF71-165FDA3608ED@gmail.com%3E
Mark's suggestion in that email regarding my original question (about prohibiting "/" in nodeNames) was that zkcontroller should replace them "/" with "_" -- but that would cause potential collisions between contexts like "/foo/solr" and "/foo_solr", so i think using something like URLEncoding makes more sense (and shouldn't impact existing ZK cluster state data for most existing users)
The attached patch enhances the test base classes to allow for randomized hostContext values, and then uses this URLEncoding logic in ZKController to build nodeNames -- and in most cases seems to work. But thinking about "_" in paths got me paranoid about explicitly testing that which is how I discovered the crufty logic in OverseerCollectionProcessor. (NOTE: you can see the obvious OverseerCollectionProcessor errors trying to talk to the wrong URL in the test logs, and they seem to explain the subsequent test failure message, but it's also possible there is a subsequent problem i haven't noticed yet)
I haven't dug into this part of the code/problem very much yet, but i *think* the right fix here is to clean this up this code so that intead of making assumptions about the node name, is uses the clusterstate to lookup the base_url from the nodeName.
Logged error (repeated for multiple shards)...
{noformat}
[junit4:junit4] 2> 204647 T33 oasc.SolrException.log SEVERE Collection createcollection of awholynewcollection_1 failed
[junit4:junit4] 2> 204686 T31 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4] 2> 204688 T33 oasc.OverseerCollectionProcessor.createCollection Create collection awholynewcollection_2 on [127.0.0.1:57855_randctxmqvf%2F_ay, 127.0.0.1:37463_randctxmqvf%2F_ay]
[junit4:junit4] 2> 204691 T33 oasc.OverseerCollectionProcessor.createCollection SEVERE Error talking to shard: 127.0.0.1:37463/randctxmqvf%2F/ay org.apache.solr.common.SolrException: Server at http://127.0.0.1:37463/randctxmqvf%2F/ay returned non ok status:404, message:Not Found
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] 2> at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:166)
[junit4:junit4] 2> at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:133)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
{noformat}
Final test failure message...
{noformat}
<testcase classname="org.apache.solr.cloud.BasicDistributedZkTest" name="testDistribSearch" time="268.833">
<failure message="Could not find new 2 slice collection called awholynewcollection_0" type="java.lang.AssertionError">java.lang.AssertionError: Could not find new 2 slice collection called awholynewcollection_0
at __randomizedtesting.SeedInfo.seed([1BD856523B97C07C:9A3ED84A4CC8A040]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.BasicDistributedZkTest.checkForCollection(BasicDistributedZkTest.java:1053)
at org.apache.solr.cloud.BasicDistributedZkTest.testCollectionsAPI(BasicDistributedZkTest.java:768)
at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:361)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:712)
{noformat}
> SolrCloud bugs when servlet context contains "/" or "_"
> -------------------------------------------------------
>
> Key: SOLR-4136
> URL: https://issues.apache.org/jira/browse/SOLR-4136
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.0
> Reporter: Hoss Man
> Assignee: Hoss Man
> Attachments: SOLR-4136.patch
>
>
> SolrCloud does not work properly with non-trivial values for "hostContext" (ie: the servlet context path). In particular...
> * Using a hostContext containing a "/" (ie: a servlet context with a subdir path, semi-common among people who organize webapps hierarchically for lod blanacer rules) is explicitly forbidden in ZkController because of how the hostContext is used to build a ZK nodeName
> * Using a hostContext containing a "_" causes problems in OverseerCollectionProcessor where it assumes all "_" characters should be converted to "/" to reconstitute a URL from nodeName (NOTE: this code specifically has a TODO to fix this, and then has a subsequent TODO about assuming "http://" labeled "this sucks")
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-4136) SolrCloud bugs when servlet context
contains "/" or "_"
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated SOLR-4136:
---------------------------
Attachment: SOLR-4136.patch
New patch...
1) implements the "less kludgy-kludge" idea i mentioned above. I put the new public method for getting the baseURL of a nodeName in SolrZkClient in case we want to change it to be a live lookup from ZK at some point (and marked it @lucene.experimental in case we want completely move/remove it at some point)
2) improved ZkController to allow both leading and trailing "/" in the hostContext -- both of which get striped off _after_ the "empty string defaults to 'solr'" logic, and _before_ the generation of the nodeName. As as result, specifying {{hostContext="/"}} in solr.xml is now a viable way to tell solr it's running in the root context. (In general i think this makes things easier to use, but
3) enhanced the hostContext randomization logic in BaseDistributedSearchTestCase so that it randomly uses: the root context (eg: "/"), one level paths (eg: "/as"), two level paths (eg: "/s/bf"), and, sometimes including an "_" character in a path (eg: "/as_", "/s_d", "/_d/bf", etc..)
4) Changed BaseDistributedSearchTestCase to us the context path randomization logic by default. This is something i had left out of my previous patch because i wa worried about users who might have their own tests extending BaseDistributedSearchTestCase that were then making assumptions about the context being "/solr" -- but i decided having more test coverage using these random context paths was more important -- we can include an upgrade note along the lines of...
{noformat}
BaseDistributedSearchTestCase now randomizes the servlet context it uses when
creating Jetty instances. Subclasses that assume a hard coded context of
"/solr" should either be fixed to use the "String context" variable, or should
take advantage of the new BaseDistributedSearchTestCase(String) constructor
to explicitly specify a fixed servlet context path.
{noformat}
5) Fixed test bugs exposed by 3 & 4 above.
--
Mark: would really appreciate it if you could take this bad-boy for a spin and let me know what you think
> SolrCloud bugs when servlet context contains "/" or "_"
> -------------------------------------------------------
>
> Key: SOLR-4136
> URL: https://issues.apache.org/jira/browse/SOLR-4136
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.0
> Reporter: Hoss Man
> Assignee: Hoss Man
> Attachments: SOLR-4136.patch, SOLR-4136.patch
>
>
> SolrCloud does not work properly with non-trivial values for "hostContext" (ie: the servlet context path). In particular...
> * Using a hostContext containing a "/" (ie: a servlet context with a subdir path, semi-common among people who organize webapps hierarchically for lod blanacer rules) is explicitly forbidden in ZkController because of how the hostContext is used to build a ZK nodeName
> * Using a hostContext containing a "\_" causes problems in OverseerCollectionProcessor where it assumes all "\_" characters should be converted to "/" to reconstitute a URL from nodeName (NOTE: this code specifically has a TODO to fix this, and then has a subsequent TODO about assuming "http://" labeled "this sucks")
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-4136) SolrCloud bugs when servlet context
contains "/" or "_"
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated SOLR-4136:
---------------------------
Description:
SolrCloud does not work properly with non-trivial values for "hostContext" (ie: the servlet context path). In particular...
* Using a hostContext containing a "/" (ie: a servlet context with a subdir path, semi-common among people who organize webapps hierarchically for lod blanacer rules) is explicitly forbidden in ZkController because of how the hostContext is used to build a ZK nodeName
* Using a hostContext containing a "\_" causes problems in OverseerCollectionProcessor where it assumes all "\_" characters should be converted to "/" to reconstitute a URL from nodeName (NOTE: this code specifically has a TODO to fix this, and then has a subsequent TODO about assuming "http://" labeled "this sucks")
was:
SolrCloud does not work properly with non-trivial values for "hostContext" (ie: the servlet context path). In particular...
* Using a hostContext containing a "/" (ie: a servlet context with a subdir path, semi-common among people who organize webapps hierarchically for lod blanacer rules) is explicitly forbidden in ZkController because of how the hostContext is used to build a ZK nodeName
* Using a hostContext containing a "_" causes problems in OverseerCollectionProcessor where it assumes all "_" characters should be converted to "/" to reconstitute a URL from nodeName (NOTE: this code specifically has a TODO to fix this, and then has a subsequent TODO about assuming "http://" labeled "this sucks")
> SolrCloud bugs when servlet context contains "/" or "_"
> -------------------------------------------------------
>
> Key: SOLR-4136
> URL: https://issues.apache.org/jira/browse/SOLR-4136
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.0
> Reporter: Hoss Man
> Assignee: Hoss Man
> Attachments: SOLR-4136.patch
>
>
> SolrCloud does not work properly with non-trivial values for "hostContext" (ie: the servlet context path). In particular...
> * Using a hostContext containing a "/" (ie: a servlet context with a subdir path, semi-common among people who organize webapps hierarchically for lod blanacer rules) is explicitly forbidden in ZkController because of how the hostContext is used to build a ZK nodeName
> * Using a hostContext containing a "\_" causes problems in OverseerCollectionProcessor where it assumes all "\_" characters should be converted to "/" to reconstitute a URL from nodeName (NOTE: this code specifically has a TODO to fix this, and then has a subsequent TODO about assuming "http://" labeled "this sucks")
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org