You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Mark Miller <ma...@gmail.com> on 2021/09/06 05:06:25 UTC
Re: SolrCloud Alone: Deprecate Standalone Mode
Embedded ZooKeeper was the plan for SolrCloud from day one. It didn’t
happen because we were waiting for dynamic zk cluster membership reconfig -
and because we ate it on ZooKeeper for 10 years, which is not very
conducive to pulling it in tight under the covers.
At this point it’s built up a great perception that it’s hassle, the
approach won’t scale, users are very familiar with its dc’s, cluster
spirals, and update cut offs.
It’s too bad because it blows these other hand rolled “built in” solutions
out of the water. It’s actually easy to setup. But I wouldn’t even buy that
from myself if Solr wasn’t doing it for me.
It also scales ridiculously. As demonstrated by Facebook and others, often
doing heavy communication than we need, 50k, over 100k nodes, should not be
any issue with Solr and ZooKeeper. It’s said to see someone look at what’s
going on today and talk about how it can’t scale like that, because it’s
B.S. Utter confusion about what has and does do and what it should and
could do.
Zookeeper is fast as hell, solid as hell, and SolrClouds needs of it are on
the lower end of the scale.
But the perception is out there. The session expirations and connection
losses, and leadership and overseer fails are the collective memory. But
FYI, non of that should even happen. It’s simply incorrect use of
ZooKeeper. Same reason it doesn’t ‘appear’ this approach would scale.
Making 200k nodes that are almost all independent is not scaling SolrCloud.
It’s a silly comparison to even think about in the same breath as
SolrCloud. It’s a silly square going after a round hole when you have
Lucene modules that lay out, extremely solidly, everything you want or need
for some faux cloud system.
But yeah, if you told me I had to setup a zk cluster before I could try a
cloud cluster, I’d never have even downloaded SolrCloud. But it’s a silly
obstacle. It’s solid as he, fast as hell, scalable as hell. Way the hell
better than the ad hoc bs out there that appears better comparing from a
Solr angle. And a user should almost never see it at all. It’s the best
horse under this thing, and if you write a proper zookeeper application,
the only time a user would hear from zookeeper is if shit was
catastrophically failing for real. Not because some network switch
somewhere got hit my an electron.
Mark
On Fri, Aug 13, 2021 at 4:31 AM Jan Høydahl <ja...@cominvent.com> wrote:
> If using the embedded for three nodes (for example) solr you will get one
> of two things out of the box:
>
> - Start zk on every one of 3 nodes, get 3 independent clusters that
> know nothing of each other
> - Start zk on one "key" node, point the other two nodes at the key
> node, get one cluster that shards can be distributed across, but if your
> key node goes down all goes down
>
> Gus, you see this behavior because our embedded ZK implementation is very
> old, and have not even attempted to use the dynamic reconfig capabilities
> of recent ZK versions.
> If we choose to revamp our embedded ZK with dynamic reconfig, then we
> could have Solr node 2 connect to node 1, and then issue a reconfiguration
> command to ZK to make them form one cluster. And a 2-node ZK cluster is now
> supported, the 2nd node will be a non-voting member or observer. Then when
> a 3rd Solr node joins, it can have its embedded ZK join and form a quorum
> with the others. We'd need some additional code somewhere (on the overseer)
> which would
> * Keep track of the Zookeeper cluster
> * Issue reconfiguration commands to ZK when Solr nodes come and go
>
> Also, the ZK_HOST config may need to be re-designed, or dynamically
> updated - without introducing the risk of split-brain. Have not looked into
> the details, but my point is that if we want this to happen, it could now
> be possible, and I would not hesitate recommending such a "managed ZK" mode
> over the current one.
>
> I have a client running SolrCloud on EC2, manually provisioned, and they'd
> like to move the cluster over to ECS (AWS's managed Container Service). But
> it is ZK that stops us doing that right now, it is simply too much hassle
> to stand up a ZK ensemble in ECS. With this embedded mode, it would be much
> simpler.
>
> Jan
>
> 12. aug. 2021 kl. 16:34 skrev Gus Heck <gu...@gmail.com>:
>
> If using the embedded for three nodes (for example) solr you will get one
> of two things out of the box:
>
> - Start zk on every one of 3 nodes, get 3 independent clusters that
> know nothing of each other
> - Start zk on one "key" node, point the other two nodes at the key
> node, get one cluster that shards can be distributed across, but if your
> key node goes down all goes down
>
> These can be fine for small, simple use cases that are not mission
> critical, and/or where queries and load are known to be well behaved and
> risk is more acceptable than. I've not tried to convince the embedded zk's
> to form a zk cluster and I don't think that that process is well
> documented, but probably one could achieve that manually with a little
> research/work, probably mostly tracking down where the zk config is landing
> and manually editing it and then ensuring somehow that those changes are
> not easily lost, but it does not happen OOTB. What I "brainstormed" in the
> prior email amounts to figuring that out, documenting it and writing code
> that can make it happen automatically. (some need for discovery/management
> too).
>
> I think the biggest thing is for folks to have clarity on what risks they
> are taking, and if they have decided to accept the risk with full
> appreciation for it, that's fine. Important to also realize that zero risk
> doesn't exist and striving for it may not make economic sense in all cases.
> Being irrationally scared of risk is also a pitfall.
>
> On Thu, Aug 12, 2021 at 7:09 AM Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> I am fully happy with something that works out of box.
>>
>>
>>
>> The main problems I see with many customers is not only the complexity of
>> setup, but also that you need to install a separate Zookeeper ensemble.
>> When you tell them: Come on, use the one provided by a solr node and you
>> are fine: “no this is not allowed, see doc xy”.
>>
>>
>>
>> So let us please simplify the recommendations: If you have one or 2 or
>> three nodes in standalone node, it is perfectly fine to use embedded
>> zookeeper. We should not overreact here. A user who used Master/Slave
>> replication is also not fully fault tolerant.
>>
>>
>>
>> I’d change the documentation to say something: “If you want to scale, use
>> a separate zookeeper ensemble with a minimum of three nodes. But for simple
>> setups just relying on the good old master/slave replication (not the
>> default solr one that distributes indexing), it is perfectly fine to use
>> embedded zookeeper (on the “master” node that holds the main index). This
>> setup is then not really different from classical master/slave replication.
>>
>>
>>
>> As said before, I am not against Solr cloud, but lets keep it simple for
>> people that want to keep it simple. I am also fine to start a single node
>> cluster with zookeeper, but this should be the embedded one (just as
>> datastore for the fake cluster). And no warnings should be printed. Maybe
>> as soon as you add too many nodes, print some warning “now it is time to
>> setup a separate zookeeper ensemble”. But, please not for 2 nodes
>> (master/slave).
>>
>>
>>
>> Also where is the problem in spawning an embedded zookeeper in every node
>> by default? Why does it need to be separated?
>>
>>
>>
>> Uwe
>>
>>
>>
>> -----
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>>
>> https://www.thetaphi.de
>>
>> eMail: uwe@thetaphi.de
>>
>>
>>
>> *From:* Jan Høydahl <ja...@cominvent.com>
>> *Sent:* Wednesday, August 11, 2021 4:27 PM
>> *To:* dev@solr.apache.org
>> *Subject:* Re: SolrCloud Alone: Deprecate Standalone Mode
>>
>>
>>
>> However, we tell people not to use the embedded ZK in production, so I’m
>> curious if that’s only because it’s a single-node ZK or if there is
>> something else about the way we’ve embedded it that we would need to change?
>>
>>
>>
>> As I recall there are several reasons. First, our embedded ZK was kind of
>> a hack with some forked code etc. Second, it is not designed to be fault
>> tolerant even if you start three solr nodes this way we cannot form a
>> quorum. And perhaps third, ZK has not been officially supported on
>> Windows.. However, I believe this is all solvable if we want to day. Not
>> saying it is easy though :)
>>
>>
>>
>> Jan
>>
>>
>>
>> 11. aug. 2021 kl. 16:17 skrev Cassandra Targett <ca...@gmail.com>:
>>
>>
>>
>> So basically the proposal would be that we use the embedded ZK to
>> automatically create a quorum via multiple nodes. That’s an interesting
>> idea.
>>
>> However, we tell people not to use the embedded ZK in production, so I’m
>> curious if that’s only because it’s a single-node ZK or if there is
>> something else about the way we’ve embedded it that we would need to change?
>>
>> I was also under the impression that beyond the complexities of ZK there
>> are still use cases that SolrCloud does not adequately support, even with
>> the addition of TLOG and PULL replicas. Does anyone have any examples of
>> those?
>>
>> I’d also like to remind folks to please not use the terminology
>> “master/slave”, we removed it from the code and documentation because it’s
>> not inclusive for our community.
>>
>> Similarly, “standalone” has always been rather imprecise - it’s not
>> “standalone”, it’s a cluster but without ZK and other automation sugar. In
>> the Ref Guide we’ve settled on “user-managed”. It sounds pedantic but it
>> matters because we should be really clear about what we’re talking about -
>> deprecating and removing the ability for a single-node Solr installation?
>> Only the mode of a non-ZK cluster? Both?
>>
>> On Aug 11, 2021, 6:39 AM -0500, Eric Pugh <
>> epugh@opensourceconnections.com>, wrote:
>>
>> For small setups I’ve used a single ZK and a single Solr node very
>> successfully, the operational benefits of all the SolrCloud API’s has been
>> fantastic.
>>
>>
>>
>> I’ve always thought that us having ZooKeeper as this “front and center”
>> requirement for SolrCloud was always a weird decision that would put off a
>> lot of folks. We don’t beat our potential users over the head with the
>> fact we use Jetty for example. It’s just part of the stack.
>>
>>
>>
>> The flow that Gus proposed should have been added to SolrCloud a long
>> time ago, how much easier would it have made all our lives! The entire
>> existence of ZooKeeper should be behind APIs and be an abstraction. We
>> should do this regardless of if deprecated standalone!
>>
>>
>>
>> Uwe, if we had what Gus proposed, but eliminate zk, would that map much
>> more to what you wanted? Here is my attempt at retelling the story that
>> Gus told, but to meet the goals of folks who might want to move to ES for
>> ease:
>>
>>
>>
>> A) Start Node 1.
>> B) Start Node 2 telling it that Node 1 exists. node 2 comes up, joins
>> network and messages “at risk for split brain”.
>> C) Start Node 3 telling it that node 1 exists. node 1, node 2, node 3 all
>> under the covers are sharing state via ZK and messages “no risk for split
>> brain"
>> D) Node 4 - like node 2 but since we have optimum quorum doesn’t add to
>> ZK (under covers, hidden from user).
>> E) Node 5 - like node 3, but since we have optimum quorum doesn’t add to
>> ZK (under covers, hidden from user).
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Aug 11, 2021, at 7:15 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>>
>>
>> Hi,
>>
>>
>>
>> most of my customers prefer standalone mode and manual replication. A lot
>> of setups, especially in Germany, are very
>>
>>
>>
>> Solr Cloud is only interesting to large customers that want to scale
>> hugely. But from what I have seen, most of those have moved to
>> Elasticsearch or Opensearch (see below). The biggest issue is always the
>> stupidness of having to maintain a separate Zookeeper cloud, which adds
>> more hardware/VMs to the game and makes the thing more complex. If you want
>> to maintain up to 4 or 6 Solr nodes with one index and a few shards, the
>> overhead by Zookeeper (you need 3 of them) is – sorry to say –
>> unmaintainable. With Elasticsearch it’s easy to setup. No dedicated
>> cloud/standalone mode. You just start a single node and test it. If it
>> works fine, you start additional nodes to form a cloud. Plain simple.
>> Config files are easy to handle, you need no ip addresses hardcoded into
>> Zookeeper nodes, it just works. If you don’t want to make people move to
>> Elasticsearch/Opensearch, make them happy with their fully controllable
>> local master/slave mode.
>>
>>
>>
>> So my strong -1 to make cloud mode the default and deprecate standalone
>> mode. Unless both is the same and works without a separate zookeeper
>> cluster, I won’t change my vote.
>>
>>
>>
>> Uwe
>>
>>
>>
>> -----
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>>
>> https://www.thetaphi.de
>>
>> eMail: uwe@thetaphi.de
>>
>>
>>
>> *From:* Gus Heck <gu...@gmail.com>
>> *Sent:* Tuesday, August 10, 2021 8:34 PM
>> *To:* dev@solr.apache.org
>> *Subject:* Re: SolrCloud Alone: Deprecate Standalone Mode
>>
>>
>>
>> Or to keep things fast without retaining all the checks, one could
>> provide slow/fast modes for test, fast requiring a local zookeeper external
>> to the tests, with the tests properly namespacing themselves... that does
>> imply reworking some tests.
>>
>>
>>
>> Now that I say the above, it would be interesting if the some of the
>> tests could (also optionally) properly isolate themselves within an
>> externally running solr (probably started via cloud.sh with the latest
>> edits. ... develop, cloud.sh, test manually, run tests against same I
>> expect that there are still tests for which that makes no sense of course.
>> This is probably a crazier idea than using an external zookeeper however,
>> where zkChroot should be sufficient to isolate things I think...
>>
>>
>>
>> -Gus
>>
>>
>>
>> On Tue, Aug 10, 2021 at 2:22 PM David Smiley <ds...@apache.org> wrote:
>>
>> Good call-out on perceived complexity due to running 3 ZK nodes. For
>> many small installations, honestly Solr's embedded ZK is fine. Also, again
>> for small installations, running ZK alongside Solr (same hardware) is
>> fine. We shouldn't needlessly shame users away from doing these things as
>> if it's irresponsible. There's a spectrum of demands on Solr from low to
>> high. Anyway, I suspect it's increasingly moot with more Docker &
>> Kubernetes being used to reduce the hassles of deploying any service (be it
>> Solr or whatever). This will only increase going forward.
>>
>>
>>
>> Even if ZK becomes the only mode, I expect many checks in our codebase
>> that conditionally check for ZK to remain. We want tests that don't care
>> about SolrCloud mode to be fast, and that means not running unnecessary
>> things like ZooKeeper.
>>
>>
>> ~ David Smiley
>>
>> Apache Lucene/Solr Search Developer
>>
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>>
>>
>>
>> On Tue, Aug 10, 2021 at 12:23 PM Gus Heck <gu...@gmail.com> wrote:
>>
>> I've met several clients who really didn't want to manage zookeeper as an
>> additional service (I've talked some into it anyway, but it was clearly a
>> key reason they hadn't started/gone cloud). I think it would be far more
>> palatable if it's all "part of solr", doesn't require plumbing the docs of
>> some other project entirely, and requires neither requisitioning additional
>> hardware nor service scripts, monitoring, support that isn't "solr"
>> support... etc... then I think that alleviates some of the pain that folks
>> in small sub-sections of moderate to large orgs feel at the idea of using
>> cloud. These folks face long procurement cycles and disaster/recovery plans
>> etc, despite often having team sizes under 20... or face having to educate
>> large IT departments into handling deployments when they themselves are new
>> (of course that's how some of them wind up hiring folks like me... but
>> that's a barrier too since that has to be approved too). Also I've met
>> folks who didn't understand that it was possible to have a 1 node "cluster"
>> with zk on the same machine, and had the impression that 5 boxes (2 solr
>> and 3 zk) were absolutely required to run cloud. Which it is of course for
>> high availability with no SPOF, but it is not required if you don't need
>> high availability.
>>
>>
>>
>> I think to sunset "user managed" we need to figure out how to self manage
>> embedded zookeepers, most particularly setup for smaller orgs or lower
>> traffic installs should look like:
>>
>>
>>
>> A) Start Node 1 with zk embedded ... if you only need one node, don't
>> want high availability etc, done.
>>
>> B) Start Node 2 telling it the zk url for node 1. node 2 comes up, offers
>> to participate in zk, but does not because that would make an even number
>>
>> C) Start Node 3 telling it the zk url for node 1. node 1 (node 2 hasn't
>> started zk) node 3 offers to participate in zk, and now with 2 offers
>> pending, both 2 and 3, get up to date on the current state and th join, now
>> the embedded zk cluster is 3 nodes, not one, and no SPOF... as they grow...
>>
>> D) Node 4 - like node 2 but can use zk url of any of 1,2,3
>>
>> E) Node 5 - like node 3, but can use zk url of any 1,2,3
>>
>>
>>
>> Obviously, features for users to set a cap the size of zk clusters, don't
>> need 49 nodes on 50 servers... , ensure they put their data in a convenient
>> place that is well documented, document how to secure the inter-node
>> connections, clarity in the admin UI of what nodes have zk etc.
>>
>>
>>
>> For this embedded zk use case we should document whatever the user needs
>> to know so they don't have to sort through docs at an entirely different
>> project not necessarily focused on the things solr users need.
>>
>>
>>
>> Certainly we would still advocate for a separate zk cluster for better
>> performance/stability. In essence a supported mode with known
>> limitations... True we have to support all THAT code instead, but the
>> available feature set becomes consistent and a bazillion checks to see if
>> we have zkStateReader (or some other sentinel for cloud mode) can
>> disappear, so probably a net gain etc.
>>
>>
>>
>> On the flip side I"ve also had the thought that cluster state management
>> should be pluggable such that if a better tool than zk, or merely an
>> "already installed" tool is available solr could use it. Without careful
>> thought everything I just said could take us in the opposite direction
>>
>>
>>
>> Maybe running zk embedded is "Solr Fog" mode :)
>>
>>
>>
>> On Mon, Aug 9, 2021 at 2:55 PM Houston Putman <ho...@gmail.com>
>> wrote:
>>
>> I agree with David that the first step would be to make SolrCloud the
>> default mode.
>>
>> I made a dev list thread about this a few months ago, but I think I
>> failed to respond at some point.
>>
>> I will get back on that and address the
>>
>>
>>
>> I also really like Mike's idea that we enable very similar use cases with
>> embedded Zookeeper's,
>>
>> if at all possible, to make the transition easy for users who want to
>> stay on the user-manager mode.
>>
>>
>>
>> Marcus, I think it would be a great idea to fix up the documentation to
>> make SolrCloud the first and most prominent mode advertised.
>>
>> Never saw your original PR, but would love to give it a look if you
>> resuscitate it at some point.
>>
>>
>>
>> - Houston
>>
>>
>>
>> On Mon, Aug 9, 2021 at 2:48 PM David Smiley <ds...@apache.org> wrote:
>>
>> Given that SolrCloud is not even the default mode, I think it is
>> premature to deprecate standalone mode. Let's do this first and maybe
>> consider deprecating standalone after some time?
>>
>>
>> ~ David Smiley
>>
>> Apache Lucene/Solr Search Developer
>>
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>>
>>
>>
>> On Mon, Aug 9, 2021 at 1:58 PM Mike Drob <md...@mdrob.com> wrote:
>>
>> Could we simulate user managed replication with an embedded zookeeper
>> on the primary and pull replicas on the followers?
>>
>> On Mon, Aug 9, 2021 at 12:56 PM Jason Gerlowski <ge...@gmail.com>
>> wrote:
>> >
>> > Hey Marcus,
>> >
>> > The places I've worked in the past have all used SolrCloud primarily
>> > so I can't speak to any specifics, but my impression from reading
>> > user-list traffic is that a sizable chunk of Solr's user base prefers
>> > "User-Managed" mode (formerly called "standalone"). Some because they
>> > don't want to manage a ZooKeeper cluster. Some because the
>> > replication model in 'user-managed' fits their needs better. Some I
>> > imagine just haven't bothered to update in many years.
>> >
>> > I'm absolutely sympathetic to efforts to streamline development and
>> > reduce collective debt, but it might be tough to displace such a big
>> > chunk of users. I'm curious what others think though. Maybe the
>> > proportion of 'user-managed' users out there is smaller than I
>> > imagine.
>> >
>> > Jason
>> >
>> > On Fri, Aug 6, 2021 at 11:59 PM Marcus Eagan <ma...@gmail.com>
>> wrote:
>> > >
>> > > Hello again,
>> > >
>> > > Has the time come for us to reduce scope to move faster and with more
>> focus? Even for those not in the cloud, SolrCloud has been the undisputed
>> performance and usability champ since version 8.0. In version 9.0, I'd like
>> to propose that the deciders in the community deprecate standalone mode in
>> favor of SolrCloud.
>> > >
>> > > There are a few drivers:
>> > >
>> > > We only need to support changes that impact SolrCloud going forward.
>> I know that this is hard to stomach. But by the time Solr reaches version
>> 10.0, everyone should have migrated to SolrCloud as there is little reason
>> to continue to run standalone.
>> > > The new features keep coming to SolrCloud, but not to standalone. You
>> can see in a few ways how I embarrassingly discovered this late one night
>> while trying out a PR. If not careful, users can accidentally start Solr in
>> standalone mode. Think of all the features that they will see documented
>> but not in their environment. What a confusing user experience?
>> > > Last but certainly not least, the number of contributors to the
>> project, and the velocity of those contributions has dropped. . It does not
>> have to be that way, though. Two ways are for the community to observe our
>> push for modernization and improved user experience. Simply eliminating the
>> need to include the -c flag in the start command would be a huge win for
>> many engineers.We should make life easier for our users as much as the
>> maintainers here. We can strive to make the upgrade process from 9 to 10
>> very simple.
>> > >
>> > > I tried to make one step in this direction last year by re-ordering
>> the README to show the Solr Cloud command before the standalone command. I
>> believe that patch died on the vine, but I would be excited to revive it to
>> document this effort when the time is appropriate.
>> > >
>> > > Reason not to do it:
>> > >
>> > > Some large company out there might view this move as introducing
>> risk. I view the risk here as negligible but I welcome any perspective
>> there.
>> > > Some things I inevitably don't know.
>> > >
>> > > What do you all think?
>> > >
>> > > Thank you all for your voluntary contributions,
>> > > --
>> > > Marcus Eagan
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> > For additional commands, e-mail: dev-help@solr.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
>>
>>
>> --
>>
>> http://www.needhamsoftware.com (work)
>>
>> http://www.the111shift.com (play)
>>
>>
>>
>>
>> --
>>
>> http://www.needhamsoftware.com (work)
>>
>> http://www.the111shift.com (play)
>>
>>
>>
>> _______________________
>>
>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC |
>> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>> <http://tinyurl.com/eric-cal>
>>
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>>
>> This e-mail and all contents, including attachments, is considered to be
>> Company Confidential unless explicitly stated otherwise, regardless
>> of whether attachments are marked as such.
>>
>>
>>
>>
>>
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
>
> --
- Mark
http://about.me/markrmiller
Re: SolrCloud Alone: Deprecate Standalone Mode
Posted by Mark Miller <ma...@gmail.com>.
Actually completing this recurring discussion / decision is actually the
first step to Solr finally moving forward as a system. It can’t decide if
it’s a cloud system or a DYI fork it, assemble it system. The organizations
that scale it for real invest hugely in forked and non cloud built custom
solutions for their use case - large efforts that have brought little back
to the project. Meanwhile the internal split brain has large
development, maintenance and growth costs. If Solr decided what it was, it
would be better at it regardless of the decision.
Given the lack of interest in putting resources into understanding how
inefficient and solid the fundamental building blocks are - even some of
the newer “solutions” attempts are improvements but but missing the
fundamental issues and still largely inefficient and and not solid enough -
investment likely better served going into understanding how Solr gets
ZooKeeper wrong and addressing that first - it seems unlikely Solr will
actually tackle being a cloud system. At the same time, those that have
made their forks and workarounds don’t want to close off their paths. At
the same time, many don’t want to give up the cloud solution. So one side
limps, the other side drowns in private and independent investment,
duplicated all over for specific use cases. One group is going to be
unhappy if one group is going to end up with something that properly moves
forward.
MRM
On Mon, Sep 6, 2021 at 12:06 AM Mark Miller <ma...@gmail.com> wrote:
> Embedded ZooKeeper was the plan for SolrCloud from day one. It didn’t
> happen because we were waiting for dynamic zk cluster membership reconfig -
> and because we ate it on ZooKeeper for 10 years, which is not very
> conducive to pulling it in tight under the covers.
>
> At this point it’s built up a great perception that it’s hassle, the
> approach won’t scale, users are very familiar with its dc’s, cluster
> spirals, and update cut offs.
>
> It’s too bad because it blows these other hand rolled “built in” solutions
> out of the water. It’s actually easy to setup. But I wouldn’t even buy that
> from myself if Solr wasn’t doing it for me.
>
> It also scales ridiculously. As demonstrated by Facebook and others, often
> doing heavy communication than we need, 50k, over 100k nodes, should not be
> any issue with Solr and ZooKeeper. It’s said to see someone look at what’s
> going on today and talk about how it can’t scale like that, because it’s
> B.S. Utter confusion about what has and does do and what it should and
> could do.
>
> Zookeeper is fast as hell, solid as hell, and SolrClouds needs of it are
> on the lower end of the scale.
>
> But the perception is out there. The session expirations and connection
> losses, and leadership and overseer fails are the collective memory. But
> FYI, non of that should even happen. It’s simply incorrect use of
> ZooKeeper. Same reason it doesn’t ‘appear’ this approach would scale.
> Making 200k nodes that are almost all independent is not scaling SolrCloud.
> It’s a silly comparison to even think about in the same breath as
> SolrCloud. It’s a silly square going after a round hole when you have
> Lucene modules that lay out, extremely solidly, everything you want or need
> for some faux cloud system.
>
> But yeah, if you told me I had to setup a zk cluster before I could try a
> cloud cluster, I’d never have even downloaded SolrCloud. But it’s a silly
> obstacle. It’s solid as he, fast as hell, scalable as hell. Way the hell
> better than the ad hoc bs out there that appears better comparing from a
> Solr angle. And a user should almost never see it at all. It’s the best
> horse under this thing, and if you write a proper zookeeper application,
> the only time a user would hear from zookeeper is if shit was
> catastrophically failing for real. Not because some network switch
> somewhere got hit my an electron.
>
> Mark
>
> On Fri, Aug 13, 2021 at 4:31 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
>> If using the embedded for three nodes (for example) solr you will get one
>> of two things out of the box:
>>
>> - Start zk on every one of 3 nodes, get 3 independent clusters that
>> know nothing of each other
>> - Start zk on one "key" node, point the other two nodes at the key
>> node, get one cluster that shards can be distributed across, but if your
>> key node goes down all goes down
>>
>> Gus, you see this behavior because our embedded ZK implementation is very
>> old, and have not even attempted to use the dynamic reconfig capabilities
>> of recent ZK versions.
>> If we choose to revamp our embedded ZK with dynamic reconfig, then we
>> could have Solr node 2 connect to node 1, and then issue a reconfiguration
>> command to ZK to make them form one cluster. And a 2-node ZK cluster is now
>> supported, the 2nd node will be a non-voting member or observer. Then when
>> a 3rd Solr node joins, it can have its embedded ZK join and form a quorum
>> with the others. We'd need some additional code somewhere (on the overseer)
>> which would
>> * Keep track of the Zookeeper cluster
>> * Issue reconfiguration commands to ZK when Solr nodes come and go
>>
>> Also, the ZK_HOST config may need to be re-designed, or dynamically
>> updated - without introducing the risk of split-brain. Have not looked into
>> the details, but my point is that if we want this to happen, it could now
>> be possible, and I would not hesitate recommending such a "managed ZK" mode
>> over the current one.
>>
>> I have a client running SolrCloud on EC2, manually provisioned, and
>> they'd like to move the cluster over to ECS (AWS's managed Container
>> Service). But it is ZK that stops us doing that right now, it is simply too
>> much hassle to stand up a ZK ensemble in ECS. With this embedded mode, it
>> would be much simpler.
>>
>> Jan
>>
>> 12. aug. 2021 kl. 16:34 skrev Gus Heck <gu...@gmail.com>:
>>
>> If using the embedded for three nodes (for example) solr you will get one
>> of two things out of the box:
>>
>> - Start zk on every one of 3 nodes, get 3 independent clusters that
>> know nothing of each other
>> - Start zk on one "key" node, point the other two nodes at the key
>> node, get one cluster that shards can be distributed across, but if your
>> key node goes down all goes down
>>
>> These can be fine for small, simple use cases that are not mission
>> critical, and/or where queries and load are known to be well behaved and
>> risk is more acceptable than. I've not tried to convince the embedded zk's
>> to form a zk cluster and I don't think that that process is well
>> documented, but probably one could achieve that manually with a little
>> research/work, probably mostly tracking down where the zk config is landing
>> and manually editing it and then ensuring somehow that those changes are
>> not easily lost, but it does not happen OOTB. What I "brainstormed" in the
>> prior email amounts to figuring that out, documenting it and writing code
>> that can make it happen automatically. (some need for discovery/management
>> too).
>>
>> I think the biggest thing is for folks to have clarity on what risks they
>> are taking, and if they have decided to accept the risk with full
>> appreciation for it, that's fine. Important to also realize that zero risk
>> doesn't exist and striving for it may not make economic sense in all cases.
>> Being irrationally scared of risk is also a pitfall.
>>
>> On Thu, Aug 12, 2021 at 7:09 AM Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>>> I am fully happy with something that works out of box.
>>>
>>>
>>>
>>> The main problems I see with many customers is not only the complexity
>>> of setup, but also that you need to install a separate Zookeeper ensemble.
>>> When you tell them: Come on, use the one provided by a solr node and you
>>> are fine: “no this is not allowed, see doc xy”.
>>>
>>>
>>>
>>> So let us please simplify the recommendations: If you have one or 2 or
>>> three nodes in standalone node, it is perfectly fine to use embedded
>>> zookeeper. We should not overreact here. A user who used Master/Slave
>>> replication is also not fully fault tolerant.
>>>
>>>
>>>
>>> I’d change the documentation to say something: “If you want to scale,
>>> use a separate zookeeper ensemble with a minimum of three nodes. But for
>>> simple setups just relying on the good old master/slave replication (not
>>> the default solr one that distributes indexing), it is perfectly fine to
>>> use embedded zookeeper (on the “master” node that holds the main index).
>>> This setup is then not really different from classical master/slave
>>> replication.
>>>
>>>
>>>
>>> As said before, I am not against Solr cloud, but lets keep it simple for
>>> people that want to keep it simple. I am also fine to start a single node
>>> cluster with zookeeper, but this should be the embedded one (just as
>>> datastore for the fake cluster). And no warnings should be printed. Maybe
>>> as soon as you add too many nodes, print some warning “now it is time to
>>> setup a separate zookeeper ensemble”. But, please not for 2 nodes
>>> (master/slave).
>>>
>>>
>>>
>>> Also where is the problem in spawning an embedded zookeeper in every
>>> node by default? Why does it need to be separated?
>>>
>>>
>>>
>>> Uwe
>>>
>>>
>>>
>>> -----
>>>
>>> Uwe Schindler
>>>
>>> Achterdiek 19, D-28357 Bremen
>>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>>>
>>> https://www.thetaphi.de
>>>
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>>
>>> *From:* Jan Høydahl <ja...@cominvent.com>
>>> *Sent:* Wednesday, August 11, 2021 4:27 PM
>>> *To:* dev@solr.apache.org
>>> *Subject:* Re: SolrCloud Alone: Deprecate Standalone Mode
>>>
>>>
>>>
>>> However, we tell people not to use the embedded ZK in production, so I’m
>>> curious if that’s only because it’s a single-node ZK or if there is
>>> something else about the way we’ve embedded it that we would need to change?
>>>
>>>
>>>
>>> As I recall there are several reasons. First, our embedded ZK was kind
>>> of a hack with some forked code etc. Second, it is not designed to be fault
>>> tolerant even if you start three solr nodes this way we cannot form a
>>> quorum. And perhaps third, ZK has not been officially supported on
>>> Windows.. However, I believe this is all solvable if we want to day. Not
>>> saying it is easy though :)
>>>
>>>
>>>
>>> Jan
>>>
>>>
>>>
>>> 11. aug. 2021 kl. 16:17 skrev Cassandra Targett <ca...@gmail.com>:
>>>
>>>
>>>
>>> So basically the proposal would be that we use the embedded ZK to
>>> automatically create a quorum via multiple nodes. That’s an interesting
>>> idea.
>>>
>>> However, we tell people not to use the embedded ZK in production, so I’m
>>> curious if that’s only because it’s a single-node ZK or if there is
>>> something else about the way we’ve embedded it that we would need to change?
>>>
>>> I was also under the impression that beyond the complexities of ZK there
>>> are still use cases that SolrCloud does not adequately support, even with
>>> the addition of TLOG and PULL replicas. Does anyone have any examples of
>>> those?
>>>
>>> I’d also like to remind folks to please not use the terminology
>>> “master/slave”, we removed it from the code and documentation because it’s
>>> not inclusive for our community.
>>>
>>> Similarly, “standalone” has always been rather imprecise - it’s not
>>> “standalone”, it’s a cluster but without ZK and other automation sugar. In
>>> the Ref Guide we’ve settled on “user-managed”. It sounds pedantic but it
>>> matters because we should be really clear about what we’re talking about -
>>> deprecating and removing the ability for a single-node Solr installation?
>>> Only the mode of a non-ZK cluster? Both?
>>>
>>> On Aug 11, 2021, 6:39 AM -0500, Eric Pugh <
>>> epugh@opensourceconnections.com>, wrote:
>>>
>>> For small setups I’ve used a single ZK and a single Solr node very
>>> successfully, the operational benefits of all the SolrCloud API’s has been
>>> fantastic.
>>>
>>>
>>>
>>> I’ve always thought that us having ZooKeeper as this “front and center”
>>> requirement for SolrCloud was always a weird decision that would put off a
>>> lot of folks. We don’t beat our potential users over the head with the
>>> fact we use Jetty for example. It’s just part of the stack.
>>>
>>>
>>>
>>> The flow that Gus proposed should have been added to SolrCloud a long
>>> time ago, how much easier would it have made all our lives! The entire
>>> existence of ZooKeeper should be behind APIs and be an abstraction. We
>>> should do this regardless of if deprecated standalone!
>>>
>>>
>>>
>>> Uwe, if we had what Gus proposed, but eliminate zk, would that map much
>>> more to what you wanted? Here is my attempt at retelling the story that
>>> Gus told, but to meet the goals of folks who might want to move to ES for
>>> ease:
>>>
>>>
>>>
>>> A) Start Node 1.
>>> B) Start Node 2 telling it that Node 1 exists. node 2 comes up, joins
>>> network and messages “at risk for split brain”.
>>> C) Start Node 3 telling it that node 1 exists. node 1, node 2, node 3
>>> all under the covers are sharing state via ZK and messages “no risk for
>>> split brain"
>>> D) Node 4 - like node 2 but since we have optimum quorum doesn’t add to
>>> ZK (under covers, hidden from user).
>>> E) Node 5 - like node 3, but since we have optimum quorum doesn’t add to
>>> ZK (under covers, hidden from user).
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Aug 11, 2021, at 7:15 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> most of my customers prefer standalone mode and manual replication. A
>>> lot of setups, especially in Germany, are very
>>>
>>>
>>>
>>> Solr Cloud is only interesting to large customers that want to scale
>>> hugely. But from what I have seen, most of those have moved to
>>> Elasticsearch or Opensearch (see below). The biggest issue is always the
>>> stupidness of having to maintain a separate Zookeeper cloud, which adds
>>> more hardware/VMs to the game and makes the thing more complex. If you want
>>> to maintain up to 4 or 6 Solr nodes with one index and a few shards, the
>>> overhead by Zookeeper (you need 3 of them) is – sorry to say –
>>> unmaintainable. With Elasticsearch it’s easy to setup. No dedicated
>>> cloud/standalone mode. You just start a single node and test it. If it
>>> works fine, you start additional nodes to form a cloud. Plain simple.
>>> Config files are easy to handle, you need no ip addresses hardcoded into
>>> Zookeeper nodes, it just works. If you don’t want to make people move to
>>> Elasticsearch/Opensearch, make them happy with their fully controllable
>>> local master/slave mode.
>>>
>>>
>>>
>>> So my strong -1 to make cloud mode the default and deprecate standalone
>>> mode. Unless both is the same and works without a separate zookeeper
>>> cluster, I won’t change my vote.
>>>
>>>
>>>
>>> Uwe
>>>
>>>
>>>
>>> -----
>>>
>>> Uwe Schindler
>>>
>>> Achterdiek 19, D-28357 Bremen
>>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>>>
>>> https://www.thetaphi.de
>>>
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>>
>>> *From:* Gus Heck <gu...@gmail.com>
>>> *Sent:* Tuesday, August 10, 2021 8:34 PM
>>> *To:* dev@solr.apache.org
>>> *Subject:* Re: SolrCloud Alone: Deprecate Standalone Mode
>>>
>>>
>>>
>>> Or to keep things fast without retaining all the checks, one could
>>> provide slow/fast modes for test, fast requiring a local zookeeper external
>>> to the tests, with the tests properly namespacing themselves... that does
>>> imply reworking some tests.
>>>
>>>
>>>
>>> Now that I say the above, it would be interesting if the some of the
>>> tests could (also optionally) properly isolate themselves within an
>>> externally running solr (probably started via cloud.sh with the latest
>>> edits. ... develop, cloud.sh, test manually, run tests against same I
>>> expect that there are still tests for which that makes no sense of course.
>>> This is probably a crazier idea than using an external zookeeper however,
>>> where zkChroot should be sufficient to isolate things I think...
>>>
>>>
>>>
>>> -Gus
>>>
>>>
>>>
>>> On Tue, Aug 10, 2021 at 2:22 PM David Smiley <ds...@apache.org> wrote:
>>>
>>> Good call-out on perceived complexity due to running 3 ZK nodes. For
>>> many small installations, honestly Solr's embedded ZK is fine. Also, again
>>> for small installations, running ZK alongside Solr (same hardware) is
>>> fine. We shouldn't needlessly shame users away from doing these things as
>>> if it's irresponsible. There's a spectrum of demands on Solr from low to
>>> high. Anyway, I suspect it's increasingly moot with more Docker &
>>> Kubernetes being used to reduce the hassles of deploying any service (be it
>>> Solr or whatever). This will only increase going forward.
>>>
>>>
>>>
>>> Even if ZK becomes the only mode, I expect many checks in our codebase
>>> that conditionally check for ZK to remain. We want tests that don't care
>>> about SolrCloud mode to be fast, and that means not running unnecessary
>>> things like ZooKeeper.
>>>
>>>
>>> ~ David Smiley
>>>
>>> Apache Lucene/Solr Search Developer
>>>
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Aug 10, 2021 at 12:23 PM Gus Heck <gu...@gmail.com> wrote:
>>>
>>> I've met several clients who really didn't want to manage zookeeper as
>>> an additional service (I've talked some into it anyway, but it was clearly
>>> a key reason they hadn't started/gone cloud). I think it would be far more
>>> palatable if it's all "part of solr", doesn't require plumbing the docs of
>>> some other project entirely, and requires neither requisitioning additional
>>> hardware nor service scripts, monitoring, support that isn't "solr"
>>> support... etc... then I think that alleviates some of the pain that folks
>>> in small sub-sections of moderate to large orgs feel at the idea of using
>>> cloud. These folks face long procurement cycles and disaster/recovery plans
>>> etc, despite often having team sizes under 20... or face having to educate
>>> large IT departments into handling deployments when they themselves are new
>>> (of course that's how some of them wind up hiring folks like me... but
>>> that's a barrier too since that has to be approved too). Also I've met
>>> folks who didn't understand that it was possible to have a 1 node "cluster"
>>> with zk on the same machine, and had the impression that 5 boxes (2 solr
>>> and 3 zk) were absolutely required to run cloud. Which it is of course for
>>> high availability with no SPOF, but it is not required if you don't need
>>> high availability.
>>>
>>>
>>>
>>> I think to sunset "user managed" we need to figure out how to self
>>> manage embedded zookeepers, most particularly setup for smaller orgs or
>>> lower traffic installs should look like:
>>>
>>>
>>>
>>> A) Start Node 1 with zk embedded ... if you only need one node, don't
>>> want high availability etc, done.
>>>
>>> B) Start Node 2 telling it the zk url for node 1. node 2 comes up,
>>> offers to participate in zk, but does not because that would make an even
>>> number
>>>
>>> C) Start Node 3 telling it the zk url for node 1. node 1 (node 2 hasn't
>>> started zk) node 3 offers to participate in zk, and now with 2 offers
>>> pending, both 2 and 3, get up to date on the current state and th join, now
>>> the embedded zk cluster is 3 nodes, not one, and no SPOF... as they grow...
>>>
>>> D) Node 4 - like node 2 but can use zk url of any of 1,2,3
>>>
>>> E) Node 5 - like node 3, but can use zk url of any 1,2,3
>>>
>>>
>>>
>>> Obviously, features for users to set a cap the size of zk clusters,
>>> don't need 49 nodes on 50 servers... , ensure they put their data in a
>>> convenient place that is well documented, document how to secure the
>>> inter-node connections, clarity in the admin UI of what nodes have zk etc.
>>>
>>>
>>>
>>> For this embedded zk use case we should document whatever the user needs
>>> to know so they don't have to sort through docs at an entirely different
>>> project not necessarily focused on the things solr users need.
>>>
>>>
>>>
>>> Certainly we would still advocate for a separate zk cluster for better
>>> performance/stability. In essence a supported mode with known
>>> limitations... True we have to support all THAT code instead, but the
>>> available feature set becomes consistent and a bazillion checks to see if
>>> we have zkStateReader (or some other sentinel for cloud mode) can
>>> disappear, so probably a net gain etc.
>>>
>>>
>>>
>>> On the flip side I"ve also had the thought that cluster state management
>>> should be pluggable such that if a better tool than zk, or merely an
>>> "already installed" tool is available solr could use it. Without careful
>>> thought everything I just said could take us in the opposite direction
>>>
>>>
>>>
>>> Maybe running zk embedded is "Solr Fog" mode :)
>>>
>>>
>>>
>>> On Mon, Aug 9, 2021 at 2:55 PM Houston Putman <ho...@gmail.com>
>>> wrote:
>>>
>>> I agree with David that the first step would be to make SolrCloud the
>>> default mode.
>>>
>>> I made a dev list thread about this a few months ago, but I think I
>>> failed to respond at some point.
>>>
>>> I will get back on that and address the
>>>
>>>
>>>
>>> I also really like Mike's idea that we enable very similar use cases
>>> with embedded Zookeeper's,
>>>
>>> if at all possible, to make the transition easy for users who want to
>>> stay on the user-manager mode.
>>>
>>>
>>>
>>> Marcus, I think it would be a great idea to fix up the documentation to
>>> make SolrCloud the first and most prominent mode advertised.
>>>
>>> Never saw your original PR, but would love to give it a look if you
>>> resuscitate it at some point.
>>>
>>>
>>>
>>> - Houston
>>>
>>>
>>>
>>> On Mon, Aug 9, 2021 at 2:48 PM David Smiley <ds...@apache.org> wrote:
>>>
>>> Given that SolrCloud is not even the default mode, I think it is
>>> premature to deprecate standalone mode. Let's do this first and maybe
>>> consider deprecating standalone after some time?
>>>
>>>
>>> ~ David Smiley
>>>
>>> Apache Lucene/Solr Search Developer
>>>
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Aug 9, 2021 at 1:58 PM Mike Drob <md...@mdrob.com> wrote:
>>>
>>> Could we simulate user managed replication with an embedded zookeeper
>>> on the primary and pull replicas on the followers?
>>>
>>> On Mon, Aug 9, 2021 at 12:56 PM Jason Gerlowski <ge...@gmail.com>
>>> wrote:
>>> >
>>> > Hey Marcus,
>>> >
>>> > The places I've worked in the past have all used SolrCloud primarily
>>> > so I can't speak to any specifics, but my impression from reading
>>> > user-list traffic is that a sizable chunk of Solr's user base prefers
>>> > "User-Managed" mode (formerly called "standalone"). Some because they
>>> > don't want to manage a ZooKeeper cluster. Some because the
>>> > replication model in 'user-managed' fits their needs better. Some I
>>> > imagine just haven't bothered to update in many years.
>>> >
>>> > I'm absolutely sympathetic to efforts to streamline development and
>>> > reduce collective debt, but it might be tough to displace such a big
>>> > chunk of users. I'm curious what others think though. Maybe the
>>> > proportion of 'user-managed' users out there is smaller than I
>>> > imagine.
>>> >
>>> > Jason
>>> >
>>> > On Fri, Aug 6, 2021 at 11:59 PM Marcus Eagan <ma...@gmail.com>
>>> wrote:
>>> > >
>>> > > Hello again,
>>> > >
>>> > > Has the time come for us to reduce scope to move faster and with
>>> more focus? Even for those not in the cloud, SolrCloud has been the
>>> undisputed performance and usability champ since version 8.0. In version
>>> 9.0, I'd like to propose that the deciders in the community deprecate
>>> standalone mode in favor of SolrCloud.
>>> > >
>>> > > There are a few drivers:
>>> > >
>>> > > We only need to support changes that impact SolrCloud going forward.
>>> I know that this is hard to stomach. But by the time Solr reaches version
>>> 10.0, everyone should have migrated to SolrCloud as there is little reason
>>> to continue to run standalone.
>>> > > The new features keep coming to SolrCloud, but not to standalone.
>>> You can see in a few ways how I embarrassingly discovered this late one
>>> night while trying out a PR. If not careful, users can accidentally start
>>> Solr in standalone mode. Think of all the features that they will see
>>> documented but not in their environment. What a confusing user experience?
>>> > > Last but certainly not least, the number of contributors to the
>>> project, and the velocity of those contributions has dropped. . It does not
>>> have to be that way, though. Two ways are for the community to observe our
>>> push for modernization and improved user experience. Simply eliminating the
>>> need to include the -c flag in the start command would be a huge win for
>>> many engineers.We should make life easier for our users as much as the
>>> maintainers here. We can strive to make the upgrade process from 9 to 10
>>> very simple.
>>> > >
>>> > > I tried to make one step in this direction last year by re-ordering
>>> the README to show the Solr Cloud command before the standalone command. I
>>> believe that patch died on the vine, but I would be excited to revive it to
>>> document this effort when the time is appropriate.
>>> > >
>>> > > Reason not to do it:
>>> > >
>>> > > Some large company out there might view this move as introducing
>>> risk. I view the risk here as negligible but I welcome any perspective
>>> there.
>>> > > Some things I inevitably don't know.
>>> > >
>>> > > What do you all think?
>>> > >
>>> > > Thank you all for your voluntary contributions,
>>> > > --
>>> > > Marcus Eagan
>>> > >
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> > For additional commands, e-mail: dev-help@solr.apache.org
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>>
>>>
>>>
>>> --
>>>
>>> http://www.needhamsoftware.com (work)
>>>
>>> http://www.the111shift.com (play)
>>>
>>>
>>>
>>>
>>> --
>>>
>>> http://www.needhamsoftware.com (work)
>>>
>>> http://www.the111shift.com (play)
>>>
>>>
>>>
>>> _______________________
>>>
>>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC |
>>> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>>> <http://tinyurl.com/eric-cal>
>>>
>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>>>
>>> This e-mail and all contents, including attachments, is considered to be
>>> Company Confidential unless explicitly stated otherwise, regardless
>>> of whether attachments are marked as such.
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>>
>> --
> - Mark
>
> http://about.me/markrmiller
>
--
- Mark
http://about.me/markrmiller