Solr2

[Mark Miller <ma...@gmail.com>]

Now a I was saying ... SolrCloud2. Well, some people don’t like the name -
fair enough - I don’t like purple much and I didn’t invent either... so
Solr2.



I have some thoughts, a few things to offer. Yes, the guy behind a lot of
the current garbage. They asked me my thoughts on this thing on day one - I
told them I don’t have a single one, let me have a go and I’ll get back to
you. Done and done.


So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here - if I
could have the floor for just a moment. I’ve put just a little bit of time
and effort in before grabbing the mic. A modicum. My ego, my job, my
status, my history, really not part of the equation, so if I could request
a few lines before the rebuttal and correction and redirection.  No hurry
for those things, they are patient characters.


Anyway, give me a moment, and in return I’ll spare you the long winded
diversions.


Having considered Solr2 for some time, I see all kinds of roadblocks and
restraints and limitations. I see a path fraught with the potential to
mimic previous paths. And my goodness, I’m getting old. Fresh, promising
paths please. So Ive invested some time and effort to establish an escape
route from the safe, conservative hack and slash through the jungle that
gets less safe the deeper we dive. I’d simply tell you about it, but jungle
stories are all heart and no soul and we will all trade them all day.


So I’ve got some notes and code and maps and journals and crap instead.
Compiled from a couple expeditions. From before spelunking made me an old
man. So my machete isn’t solo diving any new trails, single handedly
slaying jungle cats any longer. The blade just keeps getting duller.


But I’ve got these previous materials. I’ll lay them out. Take a look. If
we can explore and discuss with just a passing courtesy of respect for our
relative time and investment and focus put in before letting loose, I’m
sure we can move beyond basic software counter meandering quickly enough to
actually enter the jungle.


I’ll try and lay out some evidence to help here. For instance, that
zookeeper is not the issue. That the overseer is not the issue. That
modules and multitudes of features are not the issue. Previous efforts are
not the issue. Object oriented development and agile frameworks are not the
issue. And yet all remain pertinent software development conversation. Bike
shed rainbows. You can make beautiful multicolored bike sheds with all of
them.


But discuss colors with someone else please. I can’t do it anymore.  But I
can discuss some jungle hack and slash momentum and trail blazing.  Give me
a moment. Time is fairly irrelevant on this dance floor. Tomorrow. Next
year. 3 years. Solr 42. It’s all the same timeline when the time comes.
Hell, my brain feels equally everywhere in all the timelines anyway. So one
moment, and I’ll point you to my speech, my journals, my old jungle dune
buggy - I did spend a moment or two once or twice preparing the damn thing
- just a moment or two, just 1, 2 or 3 times - and then the microphone is
all yours. But please, don’t talk to me about rainbows or shades of
magenta, I’m a jungle explorer, not a Sherwin Williams employee. Don’t
cover me with paint - I’m a roto spinning paint spewing machine. And I’d
love to spare you the paint ball war as well.


Links coming.


Mark



-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

As I think about the question of “What is the priority of this Solr2 stuff,
addressing the immediate issues in Solr or performance?” It triggers all
kinds of responses from me, because the two are so wrapped up.

But I’m gonna take a guess and say maybe the question is more like:

I’m using Solr now, there are these immediate issues around losing
overseers and leaders, this is obviously my concern and priority - is not
this Solr2 stuff premature in the face of the current issues, are you
suggesting that’s what I go work on instead?

So let me say, the Solr2 stuff is not a proposal to begin work on a Solr2.
It’s my research and thoughts and notes on such a thing. There is plenty
that I had to dig through and work through that can and will apply to the
current world.

So I don’t suggest you give up on solving problems that you face now, and
those issues prompted this whole journey. And also, I have thoughts and
notes on the future and where things on an iterative path start to come up
short.

I’ll try and make it clear what things might offer value short term, vs
what fuels ideas and thoughts and research about the future.

MRM
-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

Anyway, like I said, my thoughts and notes on a Solr 2 will arrive shortly.
They don’t involve release planning or branching or how one might go about
an effort larger than a typical incremental release.

For those I gave a date for those notes and comments, I apologize for being
late. Life I guess.

As far as quality vs speed, I feel they are extremely related in this case
more than most, but it’s not something I’ve had to choose between in my
research, I’m a multi tasked.  So I don’t have an opinion on it as a final
answer. Your choice. My notes will reference plenty of both.

Mark
-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

And to finish off the collection creation scale value, I don’t push to
reach there because it’s good for finding holes and races in state updates
and collection creation - though it is. It’s totally critical to all the
other work.

There is actually a lot more with potential to isolate because of it for
one, but the huge thing involves allowing all that effort and time finish
in a gold state, in a state that is not still of devilish issues. A lot of
core stuff I did - that survive and looks good in tests an led various,
there ends remaining all these pretty important issues to straighten out
and address and work through. And intense varied afford and testing -
completely unknown to me. And stuff totally outside creating collections.
But you push there, it pops out easily and reliably at the scale. So no you
see these things, it makes a lot of work that in another would be hard to
isolate more attainable, and the switch from mostly invisible  to easily
viable and reproducible makes the assessing easy in a way that perhaps they
did ghost up before in various rare areas and times - but that type of
thing is a world harder to identify and track down and verify solutions.

On Tue, Apr 13, 2021 at 8:40 PM Mark Miller <ma...@gmail.com> wrote:

> Didn’t really get to the root of the leader election fix yet either ...
>
> But a couple simple steps. One, even if you solve missing overseer leader,
> or shard , it’s a system that can fail too easily, often much too subtly
> for any notice. Have spent a lot of time improving hardening. And
> previously 5 min getting out of the box great from Curator. And it’s
> another thing you can harden like it’s a rock at low scale, that let you
> down all over at decent collection scale and load. Not so much curators
> impl, although we do still own a critical piece of it. That part I have
> been able to harden effective though. If I was stuck without impl, fairly
> soon I’d toss for a fresh perspective start. Already I’ve pretty heavily
> altered it over time.
>
> Lots of easy to things that make it much better on our best effort accept
> of it beating races and being the legit leader though, lower details and
> what not best left to the code.
>
> Major simple improvements that prevent  getting it wrong are mostly late
> in anyone addressing due to current whacky impl and code design.  But
>  large steps forward on ensuring correctness in terms of not losing a
> leader that you shouldn’t - but IMHO, very, very hard to beat having
> something like Curator take over the heavy lift part. Easy to do, faster
> and much more solid, maintained and kept up by someone else.
>
>
>
> On Tue, Apr 13, 2021 at 6:20 PM Mark Miller <ma...@gmail.com> wrote:
>
>> There is also work to do on test information, just a lot to try to cover,
>> not a deep need to delve into all the crevices. But just to kind get on
>> relative similar pages, compare goals and expectations and bars.
>>
>> In the current world, it's very hard to evaluate many things to my
>> current satisfaction, just because of the surface of things that are
>> affected and the relative low value of the tests. I've worked tremendously
>> to tons more value from the tests. And even then, they are of still limited
>> value. That's part of why its important that when you clear up all the
>> collection and state and core baggage, you move to large scale collections
>> stuff. Not just to find small holes and race conditions - but because of
>> the surprising stuff that can survive much more solid tests and supler
>> looping and hammering at low scale. The stuff you will can often and easily
>> be like, woah? How does that not pop until you push this hard? That level
>> of off? Yeesh. That's part and parcel of some of the difficulties in making
>> changes that users won't eat. You can solidify tests and do lots and lots
>> extra - and I'm mind boggled but what gets through.
>>
>> On Tue, Apr 13, 2021 at 5:34 PM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> Didn’t mean that issue - couple keyboard jams. But yeah, my last comment
>>> is an important one.
>>>
>>> I mean to and intended to get a little l, and over time much further,
>>> because any chunks or isolation is well below the intended goal and point.
>>>
>>> When I first started down this path, I was making drastic test
>>> improvements - moved into drastic production improvements. Feeling great.
>>> Just picking one thing after another as you gain momentum.
>>>
>>> That’s a world I just come back with, start shoving in.  Over time and
>>> with deeper inspection, that’s where things started to change. I have to
>>> kind of run through, get out pretext and storyline, but individual
>>> improvements and optimizations and insights or discoveries - that’s simply
>>> normal and kills to bring home. It’s much deeper in the journey that I
>>> start to realize we have some things to overcome greater than addressing
>>> the Overseer ir making things more efficient and faster. I’ve spent months
>>> and months before in wonderland of improvement. Great stage. It’s really
>>> the whole story I’m after vs any chapter or section. They are all
>>> independently lacking in what I want to convey.
>>>
>>> On Tue, Apr 13, 2021 at 3:29 PM Mark Miller <ma...@gmail.com>
>>> wrote:
>>>
>>>> Thats it for a short bit. Pulled in too many directions at the moment
>>>> as I try and take advantage of my wife’s absence to escape this current
>>>> hellish commitment deficit, def more coming, most usefully everything
>>>> already pre promised.
>>>>
>>>> But I’ll just quickly add - a solid in investment in developer
>>>> productivity and introspection and tools - all this change and forward
>>>> movement fear - that’s how you address that.
>>>>
>>>> You need to be able to click buttons and make tons happen. Click
>>>> another button and get useful summaries from all the logs that where
>>>> dumped. Flip a switch and get those summaries focused on your current focus.
>>>>
>>>> Have different metrics for development than the sea that are currently
>>>> available for users. Have all that super simple for the devs to setup and
>>>> use. So if you are working ok leader election, you can glance over and see
>>>> how many cores are made how fast. Thread stats how many cues enter zk
>>>> registration and started leader election.  Failed leader election.  How
>>>> many things happened that are part of the chain. How many cores were
>>>> created. State updates requested. Written. How many state published done.
>>>> How many leaders elected. How many parts of the leader election chain done.
>>>> Output to a modern metrics dashboard and shareable with other devs. You
>>>> need quick and targeted and summarized feedback and insight, easily
>>>> accesible introspection and summarization that comes out as fast and easy
>>>> as a human can iterate. You need it. It’s a complex, opaque, wild system.
>>>> But if you have an easy and valuable, and directable gaze into it - it’s a
>>>> simple system. With loads and loads of high value low hanging fruit that
>>>> takes no brilliance to harvest. This is how you move with momentum and
>>>> confidence and address the fears. You build the tools you need to see into
>>>> darkness and you make them efficient and easy and effective.
>>>>
>>>> On Tue, Apr 13, 2021 at 12:59 PM Mark Miller <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Sorry, have to work on my lazy iPhone typos.
>>>>>
>>>>> But also FYI, everything I’m laying out here, some of it may seem kind
>>>>> of out of scope, or rolling on a negative light, but I promise it’s just
>>>>> all part of the ramp and build of information that really is the base of
>>>>> understanding some of the paths you have to end up at to truly address even
>>>>> some of the more basic things you want to address - if you want to do it
>>>>> for real, in an intense and reliable environment.
>>>>>
>>>>> It’s not so much you have to solve and get to the bottom of everything
>>>>> to solve some of these more basic issues - depending on how well you need
>>>>> to solve them - but that the direction and paths are heavily weighted in
>>>>> potential and goodness based on all the surrounding context and issues that
>>>>> they will have to hold up against.
>>>>>
>>>>> On Tue, Apr 13, 2021 at 12:50 PM Mark Miller <ma...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> So now we start to move into the brighter side of the equation on
>>>>>> zookeeper. Again, plenty of low level and code will fill in.
>>>>>>
>>>>>> But the brighter side. And the kind of pertinent part, where it’s
>>>>>> almost impossible to assess the issues you are referring into, without
>>>>>> fairly drastic performance and efficiency improvement as well.
>>>>>>
>>>>>> And really, as I’ll get into, many ways to handle zk and all that,
>>>>>> been up and down the road, but a truly fantastic and scalable and beautiful
>>>>>> way, is just so easy and impactful and problem eliminating.
>>>>>>
>>>>>> I would essentially and easier and certainly far better path than
>>>>>> what you even get by simply using zookeeper in a more correct and common
>>>>>> manner.
>>>>>>
>>>>>> On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <ma...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> The dumping continues on Zookeeper and resources - I've got a few
>>>>>>> more to post, will get into a little on the simple, fast, correct way
>>>>>>> things are meant to regarding Zookeeper after a little more of the trouble
>>>>>>> and then jump over to something else.
>>>>>>>
>>>>>>> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <ma...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Okay, I've got a few videos to start, first one up, things will
>>>>>>>> kind of ramp a bit. Plenty of text and break out and code lined up,
>>>>>>>> being organized, edited, etc, but I'll shotgun into some overview videos in
>>>>>>>> the various parent topics before that all fills in. Short initial one in,
>>>>>>>> plenty coming on ZooKeeper and related.
>>>>>>>>
>>>>>>>> https://github.com/markrmiller/solr/issues/2
>>>>>>>>
>>>>>>>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <ma...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet
>>>>>>>>> have a defensive posture towards the state of the world,
>>>>>>>>> preconceived biases against me - let's try to trip the light fantastic and
>>>>>>>>> hard crash this thing.The slow roll I'm wrapping up can fill in.
>>>>>>>>>
>>>>>>>>> I'm gonna link bombs and jet around a bit.
>>>>>>>>>
>>>>>>>>> Let's kind of root of this at "shards with no leader, replicas not
>>>>>>>>> recovering, non functional Overseer etc"
>>>>>>>>>
>>>>>>>>> You can't really get anywhere without that at least workable. You
>>>>>>>>> can't get to scale and performance and understandability and many necessary
>>>>>>>>> improvements without it at least semi solid. You can't obtain reliability
>>>>>>>>> and greatness without it being ridiculously solid. To test and verify and
>>>>>>>>> explore that, you'll have quite a challenge avoiding joining my contest. So
>>>>>>>>> it's a good lead. But also, it's an undercurrent through the system. The
>>>>>>>>> connections and loop backs and dependencies and cycles - you can try to say
>>>>>>>>> what about this or that, but in the end we end up talking about the same
>>>>>>>>> thing. The core behavior of the whole thing, that underpins and connects
>>>>>>>>> everything. That can be addressed in many different ways, keeping things,
>>>>>>>>> removing things, changing things - you could do it many ways. "The overseer
>>>>>>>>> is not the problem", it must sound funny to you. But the Overseer as a
>>>>>>>>> concept has no problem. Even the poor current design can essentially work.
>>>>>>>>> 30 designs can work better, other designs can work too. The root issues
>>>>>>>>> around it don't stop at its borders. Anyway, let's do it. I'll dump the
>>>>>>>>> path here. Let's try to blast a litte.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>> Do you consider performance/scale of SolrCloud or corectness
>>>>>>>>>>> (shards with no leader, replicas not recovering, non functional Overseer
>>>>>>>>>>> etc) as the most important areas for improvement if we were to move to a
>>>>>>>>>>> v2?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ilan
>>>>>>>>>>>
>>>>>>>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com>
>>>>>>>>>>> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like
>>>>>>>>>>>> the name - fair enough - I don’t like purple much and I didn’t invent
>>>>>>>>>>>> either... so Solr2.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have some thoughts, a few things to offer. Yes, the guy
>>>>>>>>>>>> behind a lot of the current garbage. They asked me my thoughts on this
>>>>>>>>>>>> thing on day one - I told them I don’t have a single one, let me have a go
>>>>>>>>>>>> and I’ll get back to you. Done and done.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out
>>>>>>>>>>>> here - if I could have the floor for just a moment. I’ve put just a little
>>>>>>>>>>>> bit of time and effort in before grabbing the mic. A modicum. My ego, my
>>>>>>>>>>>> job, my status, my history, really not part of the equation, so if I could
>>>>>>>>>>>> request a few lines before the rebuttal and correction and redirection.
>>>>>>>>>>>>  No hurry for those things, they are patient characters.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Anyway, give me a moment, and in return I’ll spare you the long
>>>>>>>>>>>> winded diversions.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Having considered Solr2 for some time, I see all kinds of
>>>>>>>>>>>> roadblocks and restraints and limitations. I see a path fraught with the
>>>>>>>>>>>> potential to mimic previous paths. And my goodness, I’m getting old. Fresh,
>>>>>>>>>>>> promising paths please. So Ive invested some time and effort to establish
>>>>>>>>>>>> an escape route from the safe, conservative hack and slash through the
>>>>>>>>>>>> jungle that gets less safe the deeper we dive. I’d simply tell you about
>>>>>>>>>>>> it, but jungle stories are all heart and no soul and we will all trade them
>>>>>>>>>>>> all day.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So I’ve got some notes and code and maps and journals and crap
>>>>>>>>>>>> instead. Compiled from a couple expeditions. From before spelunking made me
>>>>>>>>>>>> an old man. So my machete isn’t solo diving any new trails, single handedly
>>>>>>>>>>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But I’ve got these previous materials. I’ll lay them out. Take
>>>>>>>>>>>> a look. If we can explore and discuss with just a passing courtesy of
>>>>>>>>>>>> respect for our relative time and investment and focus put in before
>>>>>>>>>>>> letting loose, I’m sure we can move beyond basic software counter
>>>>>>>>>>>> meandering quickly enough to actually enter the jungle.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I’ll try and lay out some evidence to help here. For instance,
>>>>>>>>>>>> that zookeeper is not the issue. That the overseer is not the issue. That
>>>>>>>>>>>> modules and multitudes of features are not the issue. Previous efforts are
>>>>>>>>>>>> not the issue. Object oriented development and agile frameworks are not the
>>>>>>>>>>>> issue. And yet all remain pertinent software development conversation. Bike
>>>>>>>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>>>>>>>>>>> them.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But discuss colors with someone else please. I can’t do it
>>>>>>>>>>>> anymore.  But I can discuss some jungle hack and slash
>>>>>>>>>>>> momentum and trail blazing.  Give me a moment. Time is fairly
>>>>>>>>>>>> irrelevant on this dance floor. Tomorrow. Next year. 3 years. Solr 42. It’s
>>>>>>>>>>>> all the same timeline when the time comes. Hell, my brain feels equally
>>>>>>>>>>>> everywhere in all the timelines anyway. So one moment, and I’ll point you
>>>>>>>>>>>> to my speech, my journals, my old jungle dune buggy - I did spend a moment
>>>>>>>>>>>> or two once or twice preparing the damn thing - just a moment or two, just
>>>>>>>>>>>> 1, 2 or 3 times - and then the microphone is all yours. But please, don’t
>>>>>>>>>>>> talk to me about rainbows or shades of magenta, I’m a jungle explorer, not
>>>>>>>>>>>> a Sherwin Williams employee. Don’t cover me with paint - I’m a roto
>>>>>>>>>>>> spinning paint spewing machine. And I’d love to spare you the paint ball
>>>>>>>>>>>> war as well.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Links coming.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Mark
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> - Mark
>>>>>>>>>>>>
>>>>>>>>>>>> http://about.me/markrmiller
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> - Mark
>>>>>>>>>>
>>>>>>>>>> http://about.me/markrmiller
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> - Mark
>>>>>>>>
>>>>>>>> http://about.me/markrmiller
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> - Mark
>>>>>>>
>>>>>>> http://about.me/markrmiller
>>>>>>>
>>>>>> --
>>>>>> - Mark
>>>>>>
>>>>>> http://about.me/markrmiller
>>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://about.me/markrmiller
>>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://about.me/markrmiller
>>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>>
>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

Didn’t really get to the root of the leader election fix yet either ...

But a couple simple steps. One, even if you solve missing overseer leader,
or shard , it’s a system that can fail too easily, often much too subtly
for any notice. Have spent a lot of time improving hardening. And
previously 5 min getting out of the box great from Curator. And it’s
another thing you can harden like it’s a rock at low scale, that let you
down all over at decent collection scale and load. Not so much curators
impl, although we do still own a critical piece of it. That part I have
been able to harden effective though. If I was stuck without impl, fairly
soon I’d toss for a fresh perspective start. Already I’ve pretty heavily
altered it over time.

Lots of easy to things that make it much better on our best effort accept
of it beating races and being the legit leader though, lower details and
what not best left to the code.

Major simple improvements that prevent  getting it wrong are mostly late in
anyone addressing due to current whacky impl and code design.  But  large
steps forward on ensuring correctness in terms of not losing a leader that
you shouldn’t - but IMHO, very, very hard to beat having something like
Curator take over the heavy lift part. Easy to do, faster and much more
solid, maintained and kept up by someone else.



On Tue, Apr 13, 2021 at 6:20 PM Mark Miller <ma...@gmail.com> wrote:

> There is also work to do on test information, just a lot to try to cover,
> not a deep need to delve into all the crevices. But just to kind get on
> relative similar pages, compare goals and expectations and bars.
>
> In the current world, it's very hard to evaluate many things to my current
> satisfaction, just because of the surface of things that are affected and
> the relative low value of the tests. I've worked tremendously to tons more
> value from the tests. And even then, they are of still limited value.
> That's part of why its important that when you clear up all the collection
> and state and core baggage, you move to large scale collections stuff. Not
> just to find small holes and race conditions - but because of the
> surprising stuff that can survive much more solid tests and supler looping
> and hammering at low scale. The stuff you will can often and easily be
> like, woah? How does that not pop until you push this hard? That level of
> off? Yeesh. That's part and parcel of some of the difficulties in making
> changes that users won't eat. You can solidify tests and do lots and lots
> extra - and I'm mind boggled but what gets through.
>
> On Tue, Apr 13, 2021 at 5:34 PM Mark Miller <ma...@gmail.com> wrote:
>
>> Didn’t mean that issue - couple keyboard jams. But yeah, my last comment
>> is an important one.
>>
>> I mean to and intended to get a little l, and over time much further,
>> because any chunks or isolation is well below the intended goal and point.
>>
>> When I first started down this path, I was making drastic test
>> improvements - moved into drastic production improvements. Feeling great.
>> Just picking one thing after another as you gain momentum.
>>
>> That’s a world I just come back with, start shoving in.  Over time and
>> with deeper inspection, that’s where things started to change. I have to
>> kind of run through, get out pretext and storyline, but individual
>> improvements and optimizations and insights or discoveries - that’s simply
>> normal and kills to bring home. It’s much deeper in the journey that I
>> start to realize we have some things to overcome greater than addressing
>> the Overseer ir making things more efficient and faster. I’ve spent months
>> and months before in wonderland of improvement. Great stage. It’s really
>> the whole story I’m after vs any chapter or section. They are all
>> independently lacking in what I want to convey.
>>
>> On Tue, Apr 13, 2021 at 3:29 PM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> Thats it for a short bit. Pulled in too many directions at the moment as
>>> I try and take advantage of my wife’s absence to escape this current
>>> hellish commitment deficit, def more coming, most usefully everything
>>> already pre promised.
>>>
>>> But I’ll just quickly add - a solid in investment in developer
>>> productivity and introspection and tools - all this change and forward
>>> movement fear - that’s how you address that.
>>>
>>> You need to be able to click buttons and make tons happen. Click another
>>> button and get useful summaries from all the logs that where dumped. Flip a
>>> switch and get those summaries focused on your current focus.
>>>
>>> Have different metrics for development than the sea that are currently
>>> available for users. Have all that super simple for the devs to setup and
>>> use. So if you are working ok leader election, you can glance over and see
>>> how many cores are made how fast. Thread stats how many cues enter zk
>>> registration and started leader election.  Failed leader election.  How
>>> many things happened that are part of the chain. How many cores were
>>> created. State updates requested. Written. How many state published done.
>>> How many leaders elected. How many parts of the leader election chain done.
>>> Output to a modern metrics dashboard and shareable with other devs. You
>>> need quick and targeted and summarized feedback and insight, easily
>>> accesible introspection and summarization that comes out as fast and easy
>>> as a human can iterate. You need it. It’s a complex, opaque, wild system.
>>> But if you have an easy and valuable, and directable gaze into it - it’s a
>>> simple system. With loads and loads of high value low hanging fruit that
>>> takes no brilliance to harvest. This is how you move with momentum and
>>> confidence and address the fears. You build the tools you need to see into
>>> darkness and you make them efficient and easy and effective.
>>>
>>> On Tue, Apr 13, 2021 at 12:59 PM Mark Miller <ma...@gmail.com>
>>> wrote:
>>>
>>>> Sorry, have to work on my lazy iPhone typos.
>>>>
>>>> But also FYI, everything I’m laying out here, some of it may seem kind
>>>> of out of scope, or rolling on a negative light, but I promise it’s just
>>>> all part of the ramp and build of information that really is the base of
>>>> understanding some of the paths you have to end up at to truly address even
>>>> some of the more basic things you want to address - if you want to do it
>>>> for real, in an intense and reliable environment.
>>>>
>>>> It’s not so much you have to solve and get to the bottom of everything
>>>> to solve some of these more basic issues - depending on how well you need
>>>> to solve them - but that the direction and paths are heavily weighted in
>>>> potential and goodness based on all the surrounding context and issues that
>>>> they will have to hold up against.
>>>>
>>>> On Tue, Apr 13, 2021 at 12:50 PM Mark Miller <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> So now we start to move into the brighter side of the equation on
>>>>> zookeeper. Again, plenty of low level and code will fill in.
>>>>>
>>>>> But the brighter side. And the kind of pertinent part, where it’s
>>>>> almost impossible to assess the issues you are referring into, without
>>>>> fairly drastic performance and efficiency improvement as well.
>>>>>
>>>>> And really, as I’ll get into, many ways to handle zk and all that,
>>>>> been up and down the road, but a truly fantastic and scalable and beautiful
>>>>> way, is just so easy and impactful and problem eliminating.
>>>>>
>>>>> I would essentially and easier and certainly far better path than what
>>>>> you even get by simply using zookeeper in a more correct and common manner.
>>>>>
>>>>> On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <ma...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> The dumping continues on Zookeeper and resources - I've got a few
>>>>>> more to post, will get into a little on the simple, fast, correct way
>>>>>> things are meant to regarding Zookeeper after a little more of the trouble
>>>>>> and then jump over to something else.
>>>>>>
>>>>>> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <ma...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Okay, I've got a few videos to start, first one up, things will kind
>>>>>>> of ramp a bit. Plenty of text and break out and code lined up,
>>>>>>> being organized, edited, etc, but I'll shotgun into some overview videos in
>>>>>>> the various parent topics before that all fills in. Short initial one in,
>>>>>>> plenty coming on ZooKeeper and related.
>>>>>>>
>>>>>>> https://github.com/markrmiller/solr/issues/2
>>>>>>>
>>>>>>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <ma...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet
>>>>>>>> have a defensive posture towards the state of the world,
>>>>>>>> preconceived biases against me - let's try to trip the light fantastic and
>>>>>>>> hard crash this thing.The slow roll I'm wrapping up can fill in.
>>>>>>>>
>>>>>>>> I'm gonna link bombs and jet around a bit.
>>>>>>>>
>>>>>>>> Let's kind of root of this at "shards with no leader, replicas not
>>>>>>>> recovering, non functional Overseer etc"
>>>>>>>>
>>>>>>>> You can't really get anywhere without that at least workable. You
>>>>>>>> can't get to scale and performance and understandability and many necessary
>>>>>>>> improvements without it at least semi solid. You can't obtain reliability
>>>>>>>> and greatness without it being ridiculously solid. To test and verify and
>>>>>>>> explore that, you'll have quite a challenge avoiding joining my contest. So
>>>>>>>> it's a good lead. But also, it's an undercurrent through the system. The
>>>>>>>> connections and loop backs and dependencies and cycles - you can try to say
>>>>>>>> what about this or that, but in the end we end up talking about the same
>>>>>>>> thing. The core behavior of the whole thing, that underpins and connects
>>>>>>>> everything. That can be addressed in many different ways, keeping things,
>>>>>>>> removing things, changing things - you could do it many ways. "The overseer
>>>>>>>> is not the problem", it must sound funny to you. But the Overseer as a
>>>>>>>> concept has no problem. Even the poor current design can essentially work.
>>>>>>>> 30 designs can work better, other designs can work too. The root issues
>>>>>>>> around it don't stop at its borders. Anyway, let's do it. I'll dump the
>>>>>>>> path here. Let's try to blast a litte.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>> Do you consider performance/scale of SolrCloud or corectness
>>>>>>>>>> (shards with no leader, replicas not recovering, non functional Overseer
>>>>>>>>>> etc) as the most important areas for improvement if we were to move to a
>>>>>>>>>> v2?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ilan
>>>>>>>>>>
>>>>>>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com>
>>>>>>>>>> a écrit :
>>>>>>>>>>
>>>>>>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like
>>>>>>>>>>> the name - fair enough - I don’t like purple much and I didn’t invent
>>>>>>>>>>> either... so Solr2.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have some thoughts, a few things to offer. Yes, the guy behind
>>>>>>>>>>> a lot of the current garbage. They asked me my thoughts on this thing on
>>>>>>>>>>> day one - I told them I don’t have a single one, let me have a go and I’ll
>>>>>>>>>>> get back to you. Done and done.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out
>>>>>>>>>>> here - if I could have the floor for just a moment. I’ve put just a little
>>>>>>>>>>> bit of time and effort in before grabbing the mic. A modicum. My ego, my
>>>>>>>>>>> job, my status, my history, really not part of the equation, so if I could
>>>>>>>>>>> request a few lines before the rebuttal and correction and redirection.
>>>>>>>>>>>  No hurry for those things, they are patient characters.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Anyway, give me a moment, and in return I’ll spare you the long
>>>>>>>>>>> winded diversions.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Having considered Solr2 for some time, I see all kinds of
>>>>>>>>>>> roadblocks and restraints and limitations. I see a path fraught with the
>>>>>>>>>>> potential to mimic previous paths. And my goodness, I’m getting old. Fresh,
>>>>>>>>>>> promising paths please. So Ive invested some time and effort to establish
>>>>>>>>>>> an escape route from the safe, conservative hack and slash through the
>>>>>>>>>>> jungle that gets less safe the deeper we dive. I’d simply tell you about
>>>>>>>>>>> it, but jungle stories are all heart and no soul and we will all trade them
>>>>>>>>>>> all day.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So I’ve got some notes and code and maps and journals and crap
>>>>>>>>>>> instead. Compiled from a couple expeditions. From before spelunking made me
>>>>>>>>>>> an old man. So my machete isn’t solo diving any new trails, single handedly
>>>>>>>>>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> But I’ve got these previous materials. I’ll lay them out. Take a
>>>>>>>>>>> look. If we can explore and discuss with just a passing courtesy of respect
>>>>>>>>>>> for our relative time and investment and focus put in before letting loose,
>>>>>>>>>>> I’m sure we can move beyond basic software counter meandering quickly
>>>>>>>>>>> enough to actually enter the jungle.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I’ll try and lay out some evidence to help here. For instance,
>>>>>>>>>>> that zookeeper is not the issue. That the overseer is not the issue. That
>>>>>>>>>>> modules and multitudes of features are not the issue. Previous efforts are
>>>>>>>>>>> not the issue. Object oriented development and agile frameworks are not the
>>>>>>>>>>> issue. And yet all remain pertinent software development conversation. Bike
>>>>>>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>>>>>>>>>> them.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> But discuss colors with someone else please. I can’t do it
>>>>>>>>>>> anymore.  But I can discuss some jungle hack and slash momentum
>>>>>>>>>>> and trail blazing.  Give me a moment. Time is fairly irrelevant
>>>>>>>>>>> on this dance floor. Tomorrow. Next year. 3 years. Solr 42. It’s all the
>>>>>>>>>>> same timeline when the time comes. Hell, my brain feels equally everywhere
>>>>>>>>>>> in all the timelines anyway. So one moment, and I’ll point you to my
>>>>>>>>>>> speech, my journals, my old jungle dune buggy - I did spend a moment or two
>>>>>>>>>>> once or twice preparing the damn thing - just a moment or two, just 1, 2 or
>>>>>>>>>>> 3 times - and then the microphone is all yours. But please, don’t talk to
>>>>>>>>>>> me about rainbows or shades of magenta, I’m a jungle explorer, not a
>>>>>>>>>>> Sherwin Williams employee. Don’t cover me with paint - I’m a roto spinning
>>>>>>>>>>> paint spewing machine. And I’d love to spare you the paint ball war as
>>>>>>>>>>> well.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Links coming.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Mark
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> - Mark
>>>>>>>>>>>
>>>>>>>>>>> http://about.me/markrmiller
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>> - Mark
>>>>>>>>>
>>>>>>>>> http://about.me/markrmiller
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> - Mark
>>>>>>>
>>>>>>> http://about.me/markrmiller
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> - Mark
>>>>>>
>>>>>> http://about.me/markrmiller
>>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://about.me/markrmiller
>>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://about.me/markrmiller
>>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
>
>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

There is also work to do on test information, just a lot to try to cover,
not a deep need to delve into all the crevices. But just to kind get on
relative similar pages, compare goals and expectations and bars.

In the current world, it's very hard to evaluate many things to my current
satisfaction, just because of the surface of things that are affected and
the relative low value of the tests. I've worked tremendously to tons more
value from the tests. And even then, they are of still limited value.
That's part of why its important that when you clear up all the collection
and state and core baggage, you move to large scale collections stuff. Not
just to find small holes and race conditions - but because of the
surprising stuff that can survive much more solid tests and supler looping
and hammering at low scale. The stuff you will can often and easily be
like, woah? How does that not pop until you push this hard? That level of
off? Yeesh. That's part and parcel of some of the difficulties in making
changes that users won't eat. You can solidify tests and do lots and lots
extra - and I'm mind boggled but what gets through.

On Tue, Apr 13, 2021 at 5:34 PM Mark Miller <ma...@gmail.com> wrote:

> Didn’t mean that issue - couple keyboard jams. But yeah, my last comment
> is an important one.
>
> I mean to and intended to get a little l, and over time much further,
> because any chunks or isolation is well below the intended goal and point.
>
> When I first started down this path, I was making drastic test
> improvements - moved into drastic production improvements. Feeling great.
> Just picking one thing after another as you gain momentum.
>
> That’s a world I just come back with, start shoving in.  Over time and
> with deeper inspection, that’s where things started to change. I have to
> kind of run through, get out pretext and storyline, but individual
> improvements and optimizations and insights or discoveries - that’s simply
> normal and kills to bring home. It’s much deeper in the journey that I
> start to realize we have some things to overcome greater than addressing
> the Overseer ir making things more efficient and faster. I’ve spent months
> and months before in wonderland of improvement. Great stage. It’s really
> the whole story I’m after vs any chapter or section. They are all
> independently lacking in what I want to convey.
>
> On Tue, Apr 13, 2021 at 3:29 PM Mark Miller <ma...@gmail.com> wrote:
>
>> Thats it for a short bit. Pulled in too many directions at the moment as
>> I try and take advantage of my wife’s absence to escape this current
>> hellish commitment deficit, def more coming, most usefully everything
>> already pre promised.
>>
>> But I’ll just quickly add - a solid in investment in developer
>> productivity and introspection and tools - all this change and forward
>> movement fear - that’s how you address that.
>>
>> You need to be able to click buttons and make tons happen. Click another
>> button and get useful summaries from all the logs that where dumped. Flip a
>> switch and get those summaries focused on your current focus.
>>
>> Have different metrics for development than the sea that are currently
>> available for users. Have all that super simple for the devs to setup and
>> use. So if you are working ok leader election, you can glance over and see
>> how many cores are made how fast. Thread stats how many cues enter zk
>> registration and started leader election.  Failed leader election.  How
>> many things happened that are part of the chain. How many cores were
>> created. State updates requested. Written. How many state published done.
>> How many leaders elected. How many parts of the leader election chain done.
>> Output to a modern metrics dashboard and shareable with other devs. You
>> need quick and targeted and summarized feedback and insight, easily
>> accesible introspection and summarization that comes out as fast and easy
>> as a human can iterate. You need it. It’s a complex, opaque, wild system.
>> But if you have an easy and valuable, and directable gaze into it - it’s a
>> simple system. With loads and loads of high value low hanging fruit that
>> takes no brilliance to harvest. This is how you move with momentum and
>> confidence and address the fears. You build the tools you need to see into
>> darkness and you make them efficient and easy and effective.
>>
>> On Tue, Apr 13, 2021 at 12:59 PM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> Sorry, have to work on my lazy iPhone typos.
>>>
>>> But also FYI, everything I’m laying out here, some of it may seem kind
>>> of out of scope, or rolling on a negative light, but I promise it’s just
>>> all part of the ramp and build of information that really is the base of
>>> understanding some of the paths you have to end up at to truly address even
>>> some of the more basic things you want to address - if you want to do it
>>> for real, in an intense and reliable environment.
>>>
>>> It’s not so much you have to solve and get to the bottom of everything
>>> to solve some of these more basic issues - depending on how well you need
>>> to solve them - but that the direction and paths are heavily weighted in
>>> potential and goodness based on all the surrounding context and issues that
>>> they will have to hold up against.
>>>
>>> On Tue, Apr 13, 2021 at 12:50 PM Mark Miller <ma...@gmail.com>
>>> wrote:
>>>
>>>> So now we start to move into the brighter side of the equation on
>>>> zookeeper. Again, plenty of low level and code will fill in.
>>>>
>>>> But the brighter side. And the kind of pertinent part, where it’s
>>>> almost impossible to assess the issues you are referring into, without
>>>> fairly drastic performance and efficiency improvement as well.
>>>>
>>>> And really, as I’ll get into, many ways to handle zk and all that, been
>>>> up and down the road, but a truly fantastic and scalable and beautiful way,
>>>> is just so easy and impactful and problem eliminating.
>>>>
>>>> I would essentially and easier and certainly far better path than what
>>>> you even get by simply using zookeeper in a more correct and common manner.
>>>>
>>>> On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> The dumping continues on Zookeeper and resources - I've got a few more
>>>>> to post, will get into a little on the simple, fast, correct way things are
>>>>> meant to regarding Zookeeper after a little more of the trouble and then
>>>>> jump over to something else.
>>>>>
>>>>> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <ma...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Okay, I've got a few videos to start, first one up, things will kind
>>>>>> of ramp a bit. Plenty of text and break out and code lined up,
>>>>>> being organized, edited, etc, but I'll shotgun into some overview videos in
>>>>>> the various parent topics before that all fills in. Short initial one in,
>>>>>> plenty coming on ZooKeeper and related.
>>>>>>
>>>>>> https://github.com/markrmiller/solr/issues/2
>>>>>>
>>>>>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <ma...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have
>>>>>>> a defensive posture towards the state of the world, preconceived biases
>>>>>>> against me - let's try to trip the light fantastic and hard crash this
>>>>>>> thing.The slow roll I'm wrapping up can fill in.
>>>>>>>
>>>>>>> I'm gonna link bombs and jet around a bit.
>>>>>>>
>>>>>>> Let's kind of root of this at "shards with no leader, replicas not
>>>>>>> recovering, non functional Overseer etc"
>>>>>>>
>>>>>>> You can't really get anywhere without that at least workable. You
>>>>>>> can't get to scale and performance and understandability and many necessary
>>>>>>> improvements without it at least semi solid. You can't obtain reliability
>>>>>>> and greatness without it being ridiculously solid. To test and verify and
>>>>>>> explore that, you'll have quite a challenge avoiding joining my contest. So
>>>>>>> it's a good lead. But also, it's an undercurrent through the system. The
>>>>>>> connections and loop backs and dependencies and cycles - you can try to say
>>>>>>> what about this or that, but in the end we end up talking about the same
>>>>>>> thing. The core behavior of the whole thing, that underpins and connects
>>>>>>> everything. That can be addressed in many different ways, keeping things,
>>>>>>> removing things, changing things - you could do it many ways. "The overseer
>>>>>>> is not the problem", it must sound funny to you. But the Overseer as a
>>>>>>> concept has no problem. Even the poor current design can essentially work.
>>>>>>> 30 designs can work better, other designs can work too. The root issues
>>>>>>> around it don't stop at its borders. Anyway, let's do it. I'll dump the
>>>>>>> path here. Let's try to blast a litte.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>> Do you consider performance/scale of SolrCloud or corectness (shards
>>>>>>>>> with no leader, replicas not recovering, non functional Overseer etc) as
>>>>>>>>> the most important areas for improvement if we were to move to a v2?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ilan
>>>>>>>>>
>>>>>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com>
>>>>>>>>> a écrit :
>>>>>>>>>
>>>>>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like
>>>>>>>>>> the name - fair enough - I don’t like purple much and I didn’t invent
>>>>>>>>>> either... so Solr2.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have some thoughts, a few things to offer. Yes, the guy behind
>>>>>>>>>> a lot of the current garbage. They asked me my thoughts on this thing on
>>>>>>>>>> day one - I told them I don’t have a single one, let me have a go and I’ll
>>>>>>>>>> get back to you. Done and done.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here
>>>>>>>>>> - if I could have the floor for just a moment. I’ve put just a little bit
>>>>>>>>>> of time and effort in before grabbing the mic. A modicum. My ego, my job,
>>>>>>>>>> my status, my history, really not part of the equation, so if I could
>>>>>>>>>> request a few lines before the rebuttal and correction and redirection.
>>>>>>>>>>  No hurry for those things, they are patient characters.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Anyway, give me a moment, and in return I’ll spare you the long
>>>>>>>>>> winded diversions.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Having considered Solr2 for some time, I see all kinds of
>>>>>>>>>> roadblocks and restraints and limitations. I see a path fraught with the
>>>>>>>>>> potential to mimic previous paths. And my goodness, I’m getting old. Fresh,
>>>>>>>>>> promising paths please. So Ive invested some time and effort to establish
>>>>>>>>>> an escape route from the safe, conservative hack and slash through the
>>>>>>>>>> jungle that gets less safe the deeper we dive. I’d simply tell you about
>>>>>>>>>> it, but jungle stories are all heart and no soul and we will all trade them
>>>>>>>>>> all day.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So I’ve got some notes and code and maps and journals and crap
>>>>>>>>>> instead. Compiled from a couple expeditions. From before spelunking made me
>>>>>>>>>> an old man. So my machete isn’t solo diving any new trails, single handedly
>>>>>>>>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> But I’ve got these previous materials. I’ll lay them out. Take a
>>>>>>>>>> look. If we can explore and discuss with just a passing courtesy of respect
>>>>>>>>>> for our relative time and investment and focus put in before letting loose,
>>>>>>>>>> I’m sure we can move beyond basic software counter meandering quickly
>>>>>>>>>> enough to actually enter the jungle.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I’ll try and lay out some evidence to help here. For instance,
>>>>>>>>>> that zookeeper is not the issue. That the overseer is not the issue. That
>>>>>>>>>> modules and multitudes of features are not the issue. Previous efforts are
>>>>>>>>>> not the issue. Object oriented development and agile frameworks are not the
>>>>>>>>>> issue. And yet all remain pertinent software development conversation. Bike
>>>>>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>>>>>>>>> them.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> But discuss colors with someone else please. I can’t do it
>>>>>>>>>> anymore.  But I can discuss some jungle hack and slash momentum
>>>>>>>>>> and trail blazing.  Give me a moment. Time is fairly irrelevant
>>>>>>>>>> on this dance floor. Tomorrow. Next year. 3 years. Solr 42. It’s all the
>>>>>>>>>> same timeline when the time comes. Hell, my brain feels equally everywhere
>>>>>>>>>> in all the timelines anyway. So one moment, and I’ll point you to my
>>>>>>>>>> speech, my journals, my old jungle dune buggy - I did spend a moment or two
>>>>>>>>>> once or twice preparing the damn thing - just a moment or two, just 1, 2 or
>>>>>>>>>> 3 times - and then the microphone is all yours. But please, don’t talk to
>>>>>>>>>> me about rainbows or shades of magenta, I’m a jungle explorer, not a
>>>>>>>>>> Sherwin Williams employee. Don’t cover me with paint - I’m a roto spinning
>>>>>>>>>> paint spewing machine. And I’d love to spare you the paint ball war as
>>>>>>>>>> well.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Links coming.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Mark
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> - Mark
>>>>>>>>>>
>>>>>>>>>> http://about.me/markrmiller
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>> - Mark
>>>>>>>>
>>>>>>>> http://about.me/markrmiller
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> - Mark
>>>>>>
>>>>>> http://about.me/markrmiller
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://about.me/markrmiller
>>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://about.me/markrmiller
>>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
> - Mark
>
> http://about.me/markrmiller
>


-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

Didn’t mean that issue - couple keyboard jams. But yeah, my last comment is
an important one.

I mean to and intended to get a little l, and over time much further,
because any chunks or isolation is well below the intended goal and point.

When I first started down this path, I was making drastic test improvements
- moved into drastic production improvements. Feeling great. Just picking
one thing after another as you gain momentum.

That’s a world I just come back with, start shoving in.  Over time and with
deeper inspection, that’s where things started to change. I have to kind of
run through, get out pretext and storyline, but individual improvements and
optimizations and insights or discoveries - that’s simply normal and kills
to bring home. It’s much deeper in the journey that I start to realize we
have some things to overcome greater than addressing the Overseer ir making
things more efficient and faster. I’ve spent months and months before in
wonderland of improvement. Great stage. It’s really the whole story I’m
after vs any chapter or section. They are all independently lacking in what
I want to convey.

On Tue, Apr 13, 2021 at 3:29 PM Mark Miller <ma...@gmail.com> wrote:

> Thats it for a short bit. Pulled in too many directions at the moment as I
> try and take advantage of my wife’s absence to escape this current hellish
> commitment deficit, def more coming, most usefully everything already pre
> promised.
>
> But I’ll just quickly add - a solid in investment in developer
> productivity and introspection and tools - all this change and forward
> movement fear - that’s how you address that.
>
> You need to be able to click buttons and make tons happen. Click another
> button and get useful summaries from all the logs that where dumped. Flip a
> switch and get those summaries focused on your current focus.
>
> Have different metrics for development than the sea that are currently
> available for users. Have all that super simple for the devs to setup and
> use. So if you are working ok leader election, you can glance over and see
> how many cores are made how fast. Thread stats how many cues enter zk
> registration and started leader election.  Failed leader election.  How
> many things happened that are part of the chain. How many cores were
> created. State updates requested. Written. How many state published done.
> How many leaders elected. How many parts of the leader election chain done.
> Output to a modern metrics dashboard and shareable with other devs. You
> need quick and targeted and summarized feedback and insight, easily
> accesible introspection and summarization that comes out as fast and easy
> as a human can iterate. You need it. It’s a complex, opaque, wild system.
> But if you have an easy and valuable, and directable gaze into it - it’s a
> simple system. With loads and loads of high value low hanging fruit that
> takes no brilliance to harvest. This is how you move with momentum and
> confidence and address the fears. You build the tools you need to see into
> darkness and you make them efficient and easy and effective.
>
> On Tue, Apr 13, 2021 at 12:59 PM Mark Miller <ma...@gmail.com>
> wrote:
>
>> Sorry, have to work on my lazy iPhone typos.
>>
>> But also FYI, everything I’m laying out here, some of it may seem kind of
>> out of scope, or rolling on a negative light, but I promise it’s just all
>> part of the ramp and build of information that really is the base of
>> understanding some of the paths you have to end up at to truly address even
>> some of the more basic things you want to address - if you want to do it
>> for real, in an intense and reliable environment.
>>
>> It’s not so much you have to solve and get to the bottom of everything to
>> solve some of these more basic issues - depending on how well you need to
>> solve them - but that the direction and paths are heavily weighted in
>> potential and goodness based on all the surrounding context and issues that
>> they will have to hold up against.
>>
>> On Tue, Apr 13, 2021 at 12:50 PM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> So now we start to move into the brighter side of the equation on
>>> zookeeper. Again, plenty of low level and code will fill in.
>>>
>>> But the brighter side. And the kind of pertinent part, where it’s almost
>>> impossible to assess the issues you are referring into, without fairly
>>> drastic performance and efficiency improvement as well.
>>>
>>> And really, as I’ll get into, many ways to handle zk and all that, been
>>> up and down the road, but a truly fantastic and scalable and beautiful way,
>>> is just so easy and impactful and problem eliminating.
>>>
>>> I would essentially and easier and certainly far better path than what
>>> you even get by simply using zookeeper in a more correct and common manner.
>>>
>>> On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <ma...@gmail.com>
>>> wrote:
>>>
>>>> The dumping continues on Zookeeper and resources - I've got a few more
>>>> to post, will get into a little on the simple, fast, correct way things are
>>>> meant to regarding Zookeeper after a little more of the trouble and then
>>>> jump over to something else.
>>>>
>>>> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Okay, I've got a few videos to start, first one up, things will kind
>>>>> of ramp a bit. Plenty of text and break out and code lined up,
>>>>> being organized, edited, etc, but I'll shotgun into some overview videos in
>>>>> the various parent topics before that all fills in. Short initial one in,
>>>>> plenty coming on ZooKeeper and related.
>>>>>
>>>>> https://github.com/markrmiller/solr/issues/2
>>>>>
>>>>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <ma...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have
>>>>>> a defensive posture towards the state of the world, preconceived biases
>>>>>> against me - let's try to trip the light fantastic and hard crash this
>>>>>> thing.The slow roll I'm wrapping up can fill in.
>>>>>>
>>>>>> I'm gonna link bombs and jet around a bit.
>>>>>>
>>>>>> Let's kind of root of this at "shards with no leader, replicas not
>>>>>> recovering, non functional Overseer etc"
>>>>>>
>>>>>> You can't really get anywhere without that at least workable. You
>>>>>> can't get to scale and performance and understandability and many necessary
>>>>>> improvements without it at least semi solid. You can't obtain reliability
>>>>>> and greatness without it being ridiculously solid. To test and verify and
>>>>>> explore that, you'll have quite a challenge avoiding joining my contest. So
>>>>>> it's a good lead. But also, it's an undercurrent through the system. The
>>>>>> connections and loop backs and dependencies and cycles - you can try to say
>>>>>> what about this or that, but in the end we end up talking about the same
>>>>>> thing. The core behavior of the whole thing, that underpins and connects
>>>>>> everything. That can be addressed in many different ways, keeping things,
>>>>>> removing things, changing things - you could do it many ways. "The overseer
>>>>>> is not the problem", it must sound funny to you. But the Overseer as a
>>>>>> concept has no problem. Even the poor current design can essentially work.
>>>>>> 30 designs can work better, other designs can work too. The root issues
>>>>>> around it don't stop at its borders. Anyway, let's do it. I'll dump the
>>>>>> path here. Let's try to blast a litte.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>> Do you consider performance/scale of SolrCloud or corectness (shards
>>>>>>>> with no leader, replicas not recovering, non functional Overseer etc) as
>>>>>>>> the most important areas for improvement if we were to move to a v2?
>>>>>>>>
>>>>>>>>
>>>>>>>> Ilan
>>>>>>>>
>>>>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com>
>>>>>>>> a écrit :
>>>>>>>>
>>>>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like
>>>>>>>>> the name - fair enough - I don’t like purple much and I didn’t invent
>>>>>>>>> either... so Solr2.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have some thoughts, a few things to offer. Yes, the guy behind a
>>>>>>>>> lot of the current garbage. They asked me my thoughts on this thing on day
>>>>>>>>> one - I told them I don’t have a single one, let me have a go and I’ll get
>>>>>>>>> back to you. Done and done.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here
>>>>>>>>> - if I could have the floor for just a moment. I’ve put just a little bit
>>>>>>>>> of time and effort in before grabbing the mic. A modicum. My ego, my job,
>>>>>>>>> my status, my history, really not part of the equation, so if I could
>>>>>>>>> request a few lines before the rebuttal and correction and redirection.
>>>>>>>>>  No hurry for those things, they are patient characters.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Anyway, give me a moment, and in return I’ll spare you the long
>>>>>>>>> winded diversions.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Having considered Solr2 for some time, I see all kinds of
>>>>>>>>> roadblocks and restraints and limitations. I see a path fraught with the
>>>>>>>>> potential to mimic previous paths. And my goodness, I’m getting old. Fresh,
>>>>>>>>> promising paths please. So Ive invested some time and effort to establish
>>>>>>>>> an escape route from the safe, conservative hack and slash through the
>>>>>>>>> jungle that gets less safe the deeper we dive. I’d simply tell you about
>>>>>>>>> it, but jungle stories are all heart and no soul and we will all trade them
>>>>>>>>> all day.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So I’ve got some notes and code and maps and journals and crap
>>>>>>>>> instead. Compiled from a couple expeditions. From before spelunking made me
>>>>>>>>> an old man. So my machete isn’t solo diving any new trails, single handedly
>>>>>>>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But I’ve got these previous materials. I’ll lay them out. Take a
>>>>>>>>> look. If we can explore and discuss with just a passing courtesy of respect
>>>>>>>>> for our relative time and investment and focus put in before letting loose,
>>>>>>>>> I’m sure we can move beyond basic software counter meandering quickly
>>>>>>>>> enough to actually enter the jungle.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I’ll try and lay out some evidence to help here. For instance,
>>>>>>>>> that zookeeper is not the issue. That the overseer is not the issue. That
>>>>>>>>> modules and multitudes of features are not the issue. Previous efforts are
>>>>>>>>> not the issue. Object oriented development and agile frameworks are not the
>>>>>>>>> issue. And yet all remain pertinent software development conversation. Bike
>>>>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>>>>>>>> them.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But discuss colors with someone else please. I can’t do it
>>>>>>>>> anymore.  But I can discuss some jungle hack and slash momentum
>>>>>>>>> and trail blazing.  Give me a moment. Time is fairly irrelevant
>>>>>>>>> on this dance floor. Tomorrow. Next year. 3 years. Solr 42. It’s all the
>>>>>>>>> same timeline when the time comes. Hell, my brain feels equally everywhere
>>>>>>>>> in all the timelines anyway. So one moment, and I’ll point you to my
>>>>>>>>> speech, my journals, my old jungle dune buggy - I did spend a moment or two
>>>>>>>>> once or twice preparing the damn thing - just a moment or two, just 1, 2 or
>>>>>>>>> 3 times - and then the microphone is all yours. But please, don’t talk to
>>>>>>>>> me about rainbows or shades of magenta, I’m a jungle explorer, not a
>>>>>>>>> Sherwin Williams employee. Don’t cover me with paint - I’m a roto spinning
>>>>>>>>> paint spewing machine. And I’d love to spare you the paint ball war as
>>>>>>>>> well.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Links coming.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Mark
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> - Mark
>>>>>>>>>
>>>>>>>>> http://about.me/markrmiller
>>>>>>>>>
>>>>>>>> --
>>>>>>> - Mark
>>>>>>>
>>>>>>> http://about.me/markrmiller
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://about.me/markrmiller
>>>>>
>>>>
>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://about.me/markrmiller
>>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

Thats it for a short bit. Pulled in too many directions at the moment as I
try and take advantage of my wife’s absence to escape this current hellish
commitment deficit, def more coming, most usefully everything already pre
promised.

But I’ll just quickly add - a solid in investment in developer productivity
and introspection and tools - all this change and forward movement fear -
that’s how you address that.

You need to be able to click buttons and make tons happen. Click another
button and get useful summaries from all the logs that where dumped. Flip a
switch and get those summaries focused on your current focus.

Have different metrics for development than the sea that are currently
available for users. Have all that super simple for the devs to setup and
use. So if you are working ok leader election, you can glance over and see
how many cores are made how fast. Thread stats how many cues enter zk
registration and started leader election.  Failed leader election.  How
many things happened that are part of the chain. How many cores were
created. State updates requested. Written. How many state published done.
How many leaders elected. How many parts of the leader election chain done.
Output to a modern metrics dashboard and shareable with other devs. You
need quick and targeted and summarized feedback and insight, easily
accesible introspection and summarization that comes out as fast and easy
as a human can iterate. You need it. It’s a complex, opaque, wild system.
But if you have an easy and valuable, and directable gaze into it - it’s a
simple system. With loads and loads of high value low hanging fruit that
takes no brilliance to harvest. This is how you move with momentum and
confidence and address the fears. You build the tools you need to see into
darkness and you make them efficient and easy and effective.

On Tue, Apr 13, 2021 at 12:59 PM Mark Miller <ma...@gmail.com> wrote:

> Sorry, have to work on my lazy iPhone typos.
>
> But also FYI, everything I’m laying out here, some of it may seem kind of
> out of scope, or rolling on a negative light, but I promise it’s just all
> part of the ramp and build of information that really is the base of
> understanding some of the paths you have to end up at to truly address even
> some of the more basic things you want to address - if you want to do it
> for real, in an intense and reliable environment.
>
> It’s not so much you have to solve and get to the bottom of everything to
> solve some of these more basic issues - depending on how well you need to
> solve them - but that the direction and paths are heavily weighted in
> potential and goodness based on all the surrounding context and issues that
> they will have to hold up against.
>
> On Tue, Apr 13, 2021 at 12:50 PM Mark Miller <ma...@gmail.com>
> wrote:
>
>> So now we start to move into the brighter side of the equation on
>> zookeeper. Again, plenty of low level and code will fill in.
>>
>> But the brighter side. And the kind of pertinent part, where it’s almost
>> impossible to assess the issues you are referring into, without fairly
>> drastic performance and efficiency improvement as well.
>>
>> And really, as I’ll get into, many ways to handle zk and all that, been
>> up and down the road, but a truly fantastic and scalable and beautiful way,
>> is just so easy and impactful and problem eliminating.
>>
>> I would essentially and easier and certainly far better path than what
>> you even get by simply using zookeeper in a more correct and common manner.
>>
>> On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> The dumping continues on Zookeeper and resources - I've got a few more
>>> to post, will get into a little on the simple, fast, correct way things are
>>> meant to regarding Zookeeper after a little more of the trouble and then
>>> jump over to something else.
>>>
>>> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <ma...@gmail.com>
>>> wrote:
>>>
>>>> Okay, I've got a few videos to start, first one up, things will kind of
>>>> ramp a bit. Plenty of text and break out and code lined up,
>>>> being organized, edited, etc, but I'll shotgun into some overview videos in
>>>> the various parent topics before that all fills in. Short initial one in,
>>>> plenty coming on ZooKeeper and related.
>>>>
>>>> https://github.com/markrmiller/solr/issues/2
>>>>
>>>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have a
>>>>> defensive posture towards the state of the world, preconceived biases
>>>>> against me - let's try to trip the light fantastic and hard crash this
>>>>> thing.The slow roll I'm wrapping up can fill in.
>>>>>
>>>>> I'm gonna link bombs and jet around a bit.
>>>>>
>>>>> Let's kind of root of this at "shards with no leader, replicas not
>>>>> recovering, non functional Overseer etc"
>>>>>
>>>>> You can't really get anywhere without that at least workable. You
>>>>> can't get to scale and performance and understandability and many necessary
>>>>> improvements without it at least semi solid. You can't obtain reliability
>>>>> and greatness without it being ridiculously solid. To test and verify and
>>>>> explore that, you'll have quite a challenge avoiding joining my contest. So
>>>>> it's a good lead. But also, it's an undercurrent through the system. The
>>>>> connections and loop backs and dependencies and cycles - you can try to say
>>>>> what about this or that, but in the end we end up talking about the same
>>>>> thing. The core behavior of the whole thing, that underpins and connects
>>>>> everything. That can be addressed in many different ways, keeping things,
>>>>> removing things, changing things - you could do it many ways. "The overseer
>>>>> is not the problem", it must sound funny to you. But the Overseer as a
>>>>> concept has no problem. Even the poor current design can essentially work.
>>>>> 30 designs can work better, other designs can work too. The root issues
>>>>> around it don't stop at its borders. Anyway, let's do it. I'll dump the
>>>>> path here. Let's try to blast a litte.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>> Do you consider performance/scale of SolrCloud or corectness (shards
>>>>>>> with no leader, replicas not recovering, non functional Overseer etc) as
>>>>>>> the most important areas for improvement if we were to move to a v2?
>>>>>>>
>>>>>>>
>>>>>>> Ilan
>>>>>>>
>>>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like the
>>>>>>>> name - fair enough - I don’t like purple much and I didn’t invent either...
>>>>>>>> so Solr2.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I have some thoughts, a few things to offer. Yes, the guy behind a
>>>>>>>> lot of the current garbage. They asked me my thoughts on this thing on day
>>>>>>>> one - I told them I don’t have a single one, let me have a go and I’ll get
>>>>>>>> back to you. Done and done.
>>>>>>>>
>>>>>>>>
>>>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here -
>>>>>>>> if I could have the floor for just a moment. I’ve put just a little bit of
>>>>>>>> time and effort in before grabbing the mic. A modicum. My ego, my job, my
>>>>>>>> status, my history, really not part of the equation, so if I could request
>>>>>>>> a few lines before the rebuttal and correction and redirection.  No
>>>>>>>> hurry for those things, they are patient characters.
>>>>>>>>
>>>>>>>>
>>>>>>>> Anyway, give me a moment, and in return I’ll spare you the long
>>>>>>>> winded diversions.
>>>>>>>>
>>>>>>>>
>>>>>>>> Having considered Solr2 for some time, I see all kinds of
>>>>>>>> roadblocks and restraints and limitations. I see a path fraught with the
>>>>>>>> potential to mimic previous paths. And my goodness, I’m getting old. Fresh,
>>>>>>>> promising paths please. So Ive invested some time and effort to establish
>>>>>>>> an escape route from the safe, conservative hack and slash through the
>>>>>>>> jungle that gets less safe the deeper we dive. I’d simply tell you about
>>>>>>>> it, but jungle stories are all heart and no soul and we will all trade them
>>>>>>>> all day.
>>>>>>>>
>>>>>>>>
>>>>>>>> So I’ve got some notes and code and maps and journals and crap
>>>>>>>> instead. Compiled from a couple expeditions. From before spelunking made me
>>>>>>>> an old man. So my machete isn’t solo diving any new trails, single handedly
>>>>>>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>>>>>>
>>>>>>>>
>>>>>>>> But I’ve got these previous materials. I’ll lay them out. Take a
>>>>>>>> look. If we can explore and discuss with just a passing courtesy of respect
>>>>>>>> for our relative time and investment and focus put in before letting loose,
>>>>>>>> I’m sure we can move beyond basic software counter meandering quickly
>>>>>>>> enough to actually enter the jungle.
>>>>>>>>
>>>>>>>>
>>>>>>>> I’ll try and lay out some evidence to help here. For instance, that
>>>>>>>> zookeeper is not the issue. That the overseer is not the issue. That
>>>>>>>> modules and multitudes of features are not the issue. Previous efforts are
>>>>>>>> not the issue. Object oriented development and agile frameworks are not the
>>>>>>>> issue. And yet all remain pertinent software development conversation. Bike
>>>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>>>>>>> them.
>>>>>>>>
>>>>>>>>
>>>>>>>> But discuss colors with someone else please. I can’t do it anymore.
>>>>>>>>  But I can discuss some jungle hack and slash momentum and trail
>>>>>>>> blazing.  Give me a moment. Time is fairly irrelevant on this
>>>>>>>> dance floor. Tomorrow. Next year. 3 years. Solr 42. It’s all the same
>>>>>>>> timeline when the time comes. Hell, my brain feels equally everywhere in
>>>>>>>> all the timelines anyway. So one moment, and I’ll point you to my speech,
>>>>>>>> my journals, my old jungle dune buggy - I did spend a moment or two once or
>>>>>>>> twice preparing the damn thing - just a moment or two, just 1, 2 or 3 times
>>>>>>>> - and then the microphone is all yours. But please, don’t talk to me about
>>>>>>>> rainbows or shades of magenta, I’m a jungle explorer, not a Sherwin
>>>>>>>> Williams employee. Don’t cover me with paint - I’m a roto spinning paint
>>>>>>>> spewing machine. And I’d love to spare you the paint ball war as well.
>>>>>>>>
>>>>>>>>
>>>>>>>> Links coming.
>>>>>>>>
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> - Mark
>>>>>>>>
>>>>>>>> http://about.me/markrmiller
>>>>>>>>
>>>>>>> --
>>>>>> - Mark
>>>>>>
>>>>>> http://about.me/markrmiller
>>>>>>
>>>>>
>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://about.me/markrmiller
>>>>
>>>
>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

Sorry, have to work on my lazy iPhone typos.

But also FYI, everything I’m laying out here, some of it may seem kind of
out of scope, or rolling on a negative light, but I promise it’s just all
part of the ramp and build of information that really is the base of
understanding some of the paths you have to end up at to truly address even
some of the more basic things you want to address - if you want to do it
for real, in an intense and reliable environment.

It’s not so much you have to solve and get to the bottom of everything to
solve some of these more basic issues - depending on how well you need to
solve them - but that the direction and paths are heavily weighted in
potential and goodness based on all the surrounding context and issues that
they will have to hold up against.

On Tue, Apr 13, 2021 at 12:50 PM Mark Miller <ma...@gmail.com> wrote:

> So now we start to move into the brighter side of the equation on
> zookeeper. Again, plenty of low level and code will fill in.
>
> But the brighter side. And the kind of pertinent part, where it’s almost
> impossible to assess the issues you are referring into, without fairly
> drastic performance and efficiency improvement as well.
>
> And really, as I’ll get into, many ways to handle zk and all that, been up
> and down the road, but a truly fantastic and scalable and beautiful way, is
> just so easy and impactful and problem eliminating.
>
> I would essentially and easier and certainly far better path than what you
> even get by simply using zookeeper in a more correct and common manner.
>
> On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <ma...@gmail.com>
> wrote:
>
>> The dumping continues on Zookeeper and resources - I've got a few more to
>> post, will get into a little on the simple, fast, correct way things are
>> meant to regarding Zookeeper after a little more of the trouble and then
>> jump over to something else.
>>
>> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> Okay, I've got a few videos to start, first one up, things will kind of
>>> ramp a bit. Plenty of text and break out and code lined up,
>>> being organized, edited, etc, but I'll shotgun into some overview videos in
>>> the various parent topics before that all fills in. Short initial one in,
>>> plenty coming on ZooKeeper and related.
>>>
>>> https://github.com/markrmiller/solr/issues/2
>>>
>>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <ma...@gmail.com>
>>> wrote:
>>>
>>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have a
>>>> defensive posture towards the state of the world, preconceived biases
>>>> against me - let's try to trip the light fantastic and hard crash this
>>>> thing.The slow roll I'm wrapping up can fill in.
>>>>
>>>> I'm gonna link bombs and jet around a bit.
>>>>
>>>> Let's kind of root of this at "shards with no leader, replicas not
>>>> recovering, non functional Overseer etc"
>>>>
>>>> You can't really get anywhere without that at least workable. You can't
>>>> get to scale and performance and understandability and many necessary
>>>> improvements without it at least semi solid. You can't obtain reliability
>>>> and greatness without it being ridiculously solid. To test and verify and
>>>> explore that, you'll have quite a challenge avoiding joining my contest. So
>>>> it's a good lead. But also, it's an undercurrent through the system. The
>>>> connections and loop backs and dependencies and cycles - you can try to say
>>>> what about this or that, but in the end we end up talking about the same
>>>> thing. The core behavior of the whole thing, that underpins and connects
>>>> everything. That can be addressed in many different ways, keeping things,
>>>> removing things, changing things - you could do it many ways. "The overseer
>>>> is not the problem", it must sound funny to you. But the Overseer as a
>>>> concept has no problem. Even the poor current design can essentially work.
>>>> 30 designs can work better, other designs can work too. The root issues
>>>> around it don't stop at its borders. Anyway, let's do it. I'll dump the
>>>> path here. Let's try to blast a litte.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com>
>>>>> wrote:
>>>>>
>>>> Do you consider performance/scale of SolrCloud or corectness (shards
>>>>>> with no leader, replicas not recovering, non functional Overseer etc) as
>>>>>> the most important areas for improvement if we were to move to a v2?
>>>>>>
>>>>>>
>>>>>> Ilan
>>>>>>
>>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com> a
>>>>>> écrit :
>>>>>>
>>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like the
>>>>>>> name - fair enough - I don’t like purple much and I didn’t invent either...
>>>>>>> so Solr2.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have some thoughts, a few things to offer. Yes, the guy behind a
>>>>>>> lot of the current garbage. They asked me my thoughts on this thing on day
>>>>>>> one - I told them I don’t have a single one, let me have a go and I’ll get
>>>>>>> back to you. Done and done.
>>>>>>>
>>>>>>>
>>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here -
>>>>>>> if I could have the floor for just a moment. I’ve put just a little bit of
>>>>>>> time and effort in before grabbing the mic. A modicum. My ego, my job, my
>>>>>>> status, my history, really not part of the equation, so if I could request
>>>>>>> a few lines before the rebuttal and correction and redirection.  No
>>>>>>> hurry for those things, they are patient characters.
>>>>>>>
>>>>>>>
>>>>>>> Anyway, give me a moment, and in return I’ll spare you the long
>>>>>>> winded diversions.
>>>>>>>
>>>>>>>
>>>>>>> Having considered Solr2 for some time, I see all kinds of roadblocks
>>>>>>> and restraints and limitations. I see a path fraught with the potential to
>>>>>>> mimic previous paths. And my goodness, I’m getting old. Fresh, promising
>>>>>>> paths please. So Ive invested some time and effort to establish an escape
>>>>>>> route from the safe, conservative hack and slash through the jungle that
>>>>>>> gets less safe the deeper we dive. I’d simply tell you about it, but jungle
>>>>>>> stories are all heart and no soul and we will all trade them all day.
>>>>>>>
>>>>>>>
>>>>>>> So I’ve got some notes and code and maps and journals and crap
>>>>>>> instead. Compiled from a couple expeditions. From before spelunking made me
>>>>>>> an old man. So my machete isn’t solo diving any new trails, single handedly
>>>>>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>>>>>
>>>>>>>
>>>>>>> But I’ve got these previous materials. I’ll lay them out. Take a
>>>>>>> look. If we can explore and discuss with just a passing courtesy of respect
>>>>>>> for our relative time and investment and focus put in before letting loose,
>>>>>>> I’m sure we can move beyond basic software counter meandering quickly
>>>>>>> enough to actually enter the jungle.
>>>>>>>
>>>>>>>
>>>>>>> I’ll try and lay out some evidence to help here. For instance, that
>>>>>>> zookeeper is not the issue. That the overseer is not the issue. That
>>>>>>> modules and multitudes of features are not the issue. Previous efforts are
>>>>>>> not the issue. Object oriented development and agile frameworks are not the
>>>>>>> issue. And yet all remain pertinent software development conversation. Bike
>>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>>>>>> them.
>>>>>>>
>>>>>>>
>>>>>>> But discuss colors with someone else please. I can’t do it anymore.
>>>>>>>  But I can discuss some jungle hack and slash momentum and trail
>>>>>>> blazing.  Give me a moment. Time is fairly irrelevant on this dance
>>>>>>> floor. Tomorrow. Next year. 3 years. Solr 42. It’s all the same timeline
>>>>>>> when the time comes. Hell, my brain feels equally everywhere in all the
>>>>>>> timelines anyway. So one moment, and I’ll point you to my speech, my
>>>>>>> journals, my old jungle dune buggy - I did spend a moment or two once or
>>>>>>> twice preparing the damn thing - just a moment or two, just 1, 2 or 3 times
>>>>>>> - and then the microphone is all yours. But please, don’t talk to me about
>>>>>>> rainbows or shades of magenta, I’m a jungle explorer, not a Sherwin
>>>>>>> Williams employee. Don’t cover me with paint - I’m a roto spinning paint
>>>>>>> spewing machine. And I’d love to spare you the paint ball war as well.
>>>>>>>
>>>>>>>
>>>>>>> Links coming.
>>>>>>>
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> - Mark
>>>>>>>
>>>>>>> http://about.me/markrmiller
>>>>>>>
>>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://about.me/markrmiller
>>>>>
>>>>
>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>>
>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

So now we start to move into the brighter side of the equation on
zookeeper. Again, plenty of low level and code will fill in.

But the brighter side. And the kind of pertinent part, where it’s almost
impossible to assess the issues you are referring into, without fairly
drastic performance and efficiency improvement as well.

And really, as I’ll get into, many ways to handle zk and all that, been up
and down the road, but a truly fantastic and scalable and beautiful way, is
just so easy and impactful and problem eliminating.

I would essentially and easier and certainly far better path than what you
even get by simply using zookeeper in a more correct and common manner.

On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <ma...@gmail.com> wrote:

> The dumping continues on Zookeeper and resources - I've got a few more to
> post, will get into a little on the simple, fast, correct way things are
> meant to regarding Zookeeper after a little more of the trouble and then
> jump over to something else.
>
> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <ma...@gmail.com> wrote:
>
>> Okay, I've got a few videos to start, first one up, things will kind of
>> ramp a bit. Plenty of text and break out and code lined up,
>> being organized, edited, etc, but I'll shotgun into some overview videos in
>> the various parent topics before that all fills in. Short initial one in,
>> plenty coming on ZooKeeper and related.
>>
>> https://github.com/markrmiller/solr/issues/2
>>
>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have a
>>> defensive posture towards the state of the world, preconceived biases
>>> against me - let's try to trip the light fantastic and hard crash this
>>> thing.The slow roll I'm wrapping up can fill in.
>>>
>>> I'm gonna link bombs and jet around a bit.
>>>
>>> Let's kind of root of this at "shards with no leader, replicas not
>>> recovering, non functional Overseer etc"
>>>
>>> You can't really get anywhere without that at least workable. You can't
>>> get to scale and performance and understandability and many necessary
>>> improvements without it at least semi solid. You can't obtain reliability
>>> and greatness without it being ridiculously solid. To test and verify and
>>> explore that, you'll have quite a challenge avoiding joining my contest. So
>>> it's a good lead. But also, it's an undercurrent through the system. The
>>> connections and loop backs and dependencies and cycles - you can try to say
>>> what about this or that, but in the end we end up talking about the same
>>> thing. The core behavior of the whole thing, that underpins and connects
>>> everything. That can be addressed in many different ways, keeping things,
>>> removing things, changing things - you could do it many ways. "The overseer
>>> is not the problem", it must sound funny to you. But the Overseer as a
>>> concept has no problem. Even the poor current design can essentially work.
>>> 30 designs can work better, other designs can work too. The root issues
>>> around it don't stop at its borders. Anyway, let's do it. I'll dump the
>>> path here. Let's try to blast a litte.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com>
>>>> wrote:
>>>>
>>> Do you consider performance/scale of SolrCloud or corectness (shards
>>>>> with no leader, replicas not recovering, non functional Overseer etc) as
>>>>> the most important areas for improvement if we were to move to a v2?
>>>>>
>>>>>
>>>>> Ilan
>>>>>
>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com> a
>>>>> écrit :
>>>>>
>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like the
>>>>>> name - fair enough - I don’t like purple much and I didn’t invent either...
>>>>>> so Solr2.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have some thoughts, a few things to offer. Yes, the guy behind a
>>>>>> lot of the current garbage. They asked me my thoughts on this thing on day
>>>>>> one - I told them I don’t have a single one, let me have a go and I’ll get
>>>>>> back to you. Done and done.
>>>>>>
>>>>>>
>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here -
>>>>>> if I could have the floor for just a moment. I’ve put just a little bit of
>>>>>> time and effort in before grabbing the mic. A modicum. My ego, my job, my
>>>>>> status, my history, really not part of the equation, so if I could request
>>>>>> a few lines before the rebuttal and correction and redirection.  No
>>>>>> hurry for those things, they are patient characters.
>>>>>>
>>>>>>
>>>>>> Anyway, give me a moment, and in return I’ll spare you the long
>>>>>> winded diversions.
>>>>>>
>>>>>>
>>>>>> Having considered Solr2 for some time, I see all kinds of roadblocks
>>>>>> and restraints and limitations. I see a path fraught with the potential to
>>>>>> mimic previous paths. And my goodness, I’m getting old. Fresh, promising
>>>>>> paths please. So Ive invested some time and effort to establish an escape
>>>>>> route from the safe, conservative hack and slash through the jungle that
>>>>>> gets less safe the deeper we dive. I’d simply tell you about it, but jungle
>>>>>> stories are all heart and no soul and we will all trade them all day.
>>>>>>
>>>>>>
>>>>>> So I’ve got some notes and code and maps and journals and crap
>>>>>> instead. Compiled from a couple expeditions. From before spelunking made me
>>>>>> an old man. So my machete isn’t solo diving any new trails, single handedly
>>>>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>>>>
>>>>>>
>>>>>> But I’ve got these previous materials. I’ll lay them out. Take a
>>>>>> look. If we can explore and discuss with just a passing courtesy of respect
>>>>>> for our relative time and investment and focus put in before letting loose,
>>>>>> I’m sure we can move beyond basic software counter meandering quickly
>>>>>> enough to actually enter the jungle.
>>>>>>
>>>>>>
>>>>>> I’ll try and lay out some evidence to help here. For instance, that
>>>>>> zookeeper is not the issue. That the overseer is not the issue. That
>>>>>> modules and multitudes of features are not the issue. Previous efforts are
>>>>>> not the issue. Object oriented development and agile frameworks are not the
>>>>>> issue. And yet all remain pertinent software development conversation. Bike
>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>>>>> them.
>>>>>>
>>>>>>
>>>>>> But discuss colors with someone else please. I can’t do it anymore.  But
>>>>>> I can discuss some jungle hack and slash momentum and trail blazing.
>>>>>>  Give me a moment. Time is fairly irrelevant on this dance floor.
>>>>>> Tomorrow. Next year. 3 years. Solr 42. It’s all the same timeline when the
>>>>>> time comes. Hell, my brain feels equally everywhere in all the timelines
>>>>>> anyway. So one moment, and I’ll point you to my speech, my journals, my old
>>>>>> jungle dune buggy - I did spend a moment or two once or twice preparing the
>>>>>> damn thing - just a moment or two, just 1, 2 or 3 times - and then the
>>>>>> microphone is all yours. But please, don’t talk to me about rainbows or
>>>>>> shades of magenta, I’m a jungle explorer, not a Sherwin Williams employee.
>>>>>> Don’t cover me with paint - I’m a roto spinning paint spewing machine. And
>>>>>> I’d love to spare you the paint ball war as well.
>>>>>>
>>>>>>
>>>>>> Links coming.
>>>>>>
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> - Mark
>>>>>>
>>>>>> http://about.me/markrmiller
>>>>>>
>>>>> --
>>>> - Mark
>>>>
>>>> http://about.me/markrmiller
>>>>
>>>
>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
>
>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

The dumping continues on Zookeeper and resources - I've got a few more to
post, will get into a little on the simple, fast, correct way things are
meant to regarding Zookeeper after a little more of the trouble and then
jump over to something else.

On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <ma...@gmail.com> wrote:

> Okay, I've got a few videos to start, first one up, things will kind of
> ramp a bit. Plenty of text and break out and code lined up,
> being organized, edited, etc, but I'll shotgun into some overview videos in
> the various parent topics before that all fills in. Short initial one in,
> plenty coming on ZooKeeper and related.
>
> https://github.com/markrmiller/solr/issues/2
>
> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <ma...@gmail.com> wrote:
>
>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have a
>> defensive posture towards the state of the world, preconceived biases
>> against me - let's try to trip the light fantastic and hard crash this
>> thing.The slow roll I'm wrapping up can fill in.
>>
>> I'm gonna link bombs and jet around a bit.
>>
>> Let's kind of root of this at "shards with no leader, replicas not
>> recovering, non functional Overseer etc"
>>
>> You can't really get anywhere without that at least workable. You can't
>> get to scale and performance and understandability and many necessary
>> improvements without it at least semi solid. You can't obtain reliability
>> and greatness without it being ridiculously solid. To test and verify and
>> explore that, you'll have quite a challenge avoiding joining my contest. So
>> it's a good lead. But also, it's an undercurrent through the system. The
>> connections and loop backs and dependencies and cycles - you can try to say
>> what about this or that, but in the end we end up talking about the same
>> thing. The core behavior of the whole thing, that underpins and connects
>> everything. That can be addressed in many different ways, keeping things,
>> removing things, changing things - you could do it many ways. "The overseer
>> is not the problem", it must sound funny to you. But the Overseer as a
>> concept has no problem. Even the poor current design can essentially work.
>> 30 designs can work better, other designs can work too. The root issues
>> around it don't stop at its borders. Anyway, let's do it. I'll dump the
>> path here. Let's try to blast a litte.
>>
>>
>>
>>
>>
>>
>>
>>>
>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com> wrote:
>>>
>> Do you consider performance/scale of SolrCloud or corectness (shards with
>>>> no leader, replicas not recovering, non functional Overseer etc) as the
>>>> most important areas for improvement if we were to move to a v2?
>>>>
>>>>
>>>> Ilan
>>>>
>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com> a
>>>> écrit :
>>>>
>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like the
>>>>> name - fair enough - I don’t like purple much and I didn’t invent either...
>>>>> so Solr2.
>>>>>
>>>>>
>>>>>
>>>>> I have some thoughts, a few things to offer. Yes, the guy behind a lot
>>>>> of the current garbage. They asked me my thoughts on this thing on day one
>>>>> - I told them I don’t have a single one, let me have a go and I’ll get back
>>>>> to you. Done and done.
>>>>>
>>>>>
>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here - if
>>>>> I could have the floor for just a moment. I’ve put just a little bit of
>>>>> time and effort in before grabbing the mic. A modicum. My ego, my job, my
>>>>> status, my history, really not part of the equation, so if I could request
>>>>> a few lines before the rebuttal and correction and redirection.  No
>>>>> hurry for those things, they are patient characters.
>>>>>
>>>>>
>>>>> Anyway, give me a moment, and in return I’ll spare you the long winded
>>>>> diversions.
>>>>>
>>>>>
>>>>> Having considered Solr2 for some time, I see all kinds of roadblocks
>>>>> and restraints and limitations. I see a path fraught with the potential to
>>>>> mimic previous paths. And my goodness, I’m getting old. Fresh, promising
>>>>> paths please. So Ive invested some time and effort to establish an escape
>>>>> route from the safe, conservative hack and slash through the jungle that
>>>>> gets less safe the deeper we dive. I’d simply tell you about it, but jungle
>>>>> stories are all heart and no soul and we will all trade them all day.
>>>>>
>>>>>
>>>>> So I’ve got some notes and code and maps and journals and crap
>>>>> instead. Compiled from a couple expeditions. From before spelunking made me
>>>>> an old man. So my machete isn’t solo diving any new trails, single handedly
>>>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>>>
>>>>>
>>>>> But I’ve got these previous materials. I’ll lay them out. Take a look.
>>>>> If we can explore and discuss with just a passing courtesy of respect for
>>>>> our relative time and investment and focus put in before letting loose, I’m
>>>>> sure we can move beyond basic software counter meandering quickly enough to
>>>>> actually enter the jungle.
>>>>>
>>>>>
>>>>> I’ll try and lay out some evidence to help here. For instance, that
>>>>> zookeeper is not the issue. That the overseer is not the issue. That
>>>>> modules and multitudes of features are not the issue. Previous efforts are
>>>>> not the issue. Object oriented development and agile frameworks are not the
>>>>> issue. And yet all remain pertinent software development conversation. Bike
>>>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>>>> them.
>>>>>
>>>>>
>>>>> But discuss colors with someone else please. I can’t do it anymore.  But
>>>>> I can discuss some jungle hack and slash momentum and trail blazing.  Give
>>>>> me a moment. Time is fairly irrelevant on this dance floor. Tomorrow. Next
>>>>> year. 3 years. Solr 42. It’s all the same timeline when the time comes.
>>>>> Hell, my brain feels equally everywhere in all the timelines anyway. So one
>>>>> moment, and I’ll point you to my speech, my journals, my old jungle dune
>>>>> buggy - I did spend a moment or two once or twice preparing the damn thing
>>>>> - just a moment or two, just 1, 2 or 3 times - and then the microphone is
>>>>> all yours. But please, don’t talk to me about rainbows or shades of
>>>>> magenta, I’m a jungle explorer, not a Sherwin Williams employee. Don’t
>>>>> cover me with paint - I’m a roto spinning paint spewing machine. And I’d
>>>>> love to spare you the paint ball war as well.
>>>>>
>>>>>
>>>>> Links coming.
>>>>>
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://about.me/markrmiller
>>>>>
>>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>>
>
> --
> - Mark
>
> http://about.me/markrmiller
>


-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

Okay, I've got a few videos to start, first one up, things will kind of
ramp a bit. Plenty of text and break out and code lined up,
being organized, edited, etc, but I'll shotgun into some overview videos in
the various parent topics before that all fills in. Short initial one in,
plenty coming on ZooKeeper and related.

https://github.com/markrmiller/solr/issues/2

On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <ma...@gmail.com> wrote:

> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have a
> defensive posture towards the state of the world, preconceived biases
> against me - let's try to trip the light fantastic and hard crash this
> thing.The slow roll I'm wrapping up can fill in.
>
> I'm gonna link bombs and jet around a bit.
>
> Let's kind of root of this at "shards with no leader, replicas not
> recovering, non functional Overseer etc"
>
> You can't really get anywhere without that at least workable. You can't
> get to scale and performance and understandability and many necessary
> improvements without it at least semi solid. You can't obtain reliability
> and greatness without it being ridiculously solid. To test and verify and
> explore that, you'll have quite a challenge avoiding joining my contest. So
> it's a good lead. But also, it's an undercurrent through the system. The
> connections and loop backs and dependencies and cycles - you can try to say
> what about this or that, but in the end we end up talking about the same
> thing. The core behavior of the whole thing, that underpins and connects
> everything. That can be addressed in many different ways, keeping things,
> removing things, changing things - you could do it many ways. "The overseer
> is not the problem", it must sound funny to you. But the Overseer as a
> concept has no problem. Even the poor current design can essentially work.
> 30 designs can work better, other designs can work too. The root issues
> around it don't stop at its borders. Anyway, let's do it. I'll dump the
> path here. Let's try to blast a litte.
>
>
>
>
>
>
>
>>
> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com> wrote:
>>
> Do you consider performance/scale of SolrCloud or corectness (shards with
>>> no leader, replicas not recovering, non functional Overseer etc) as the
>>> most important areas for improvement if we were to move to a v2?
>>>
>>>
>>> Ilan
>>>
>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com> a
>>> écrit :
>>>
>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like the
>>>> name - fair enough - I don’t like purple much and I didn’t invent either...
>>>> so Solr2.
>>>>
>>>>
>>>>
>>>> I have some thoughts, a few things to offer. Yes, the guy behind a lot
>>>> of the current garbage. They asked me my thoughts on this thing on day one
>>>> - I told them I don’t have a single one, let me have a go and I’ll get back
>>>> to you. Done and done.
>>>>
>>>>
>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here - if
>>>> I could have the floor for just a moment. I’ve put just a little bit of
>>>> time and effort in before grabbing the mic. A modicum. My ego, my job, my
>>>> status, my history, really not part of the equation, so if I could request
>>>> a few lines before the rebuttal and correction and redirection.  No
>>>> hurry for those things, they are patient characters.
>>>>
>>>>
>>>> Anyway, give me a moment, and in return I’ll spare you the long winded
>>>> diversions.
>>>>
>>>>
>>>> Having considered Solr2 for some time, I see all kinds of roadblocks
>>>> and restraints and limitations. I see a path fraught with the potential to
>>>> mimic previous paths. And my goodness, I’m getting old. Fresh, promising
>>>> paths please. So Ive invested some time and effort to establish an escape
>>>> route from the safe, conservative hack and slash through the jungle that
>>>> gets less safe the deeper we dive. I’d simply tell you about it, but jungle
>>>> stories are all heart and no soul and we will all trade them all day.
>>>>
>>>>
>>>> So I’ve got some notes and code and maps and journals and crap instead.
>>>> Compiled from a couple expeditions. From before spelunking made me an old
>>>> man. So my machete isn’t solo diving any new trails, single handedly
>>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>>
>>>>
>>>> But I’ve got these previous materials. I’ll lay them out. Take a look.
>>>> If we can explore and discuss with just a passing courtesy of respect for
>>>> our relative time and investment and focus put in before letting loose, I’m
>>>> sure we can move beyond basic software counter meandering quickly enough to
>>>> actually enter the jungle.
>>>>
>>>>
>>>> I’ll try and lay out some evidence to help here. For instance, that
>>>> zookeeper is not the issue. That the overseer is not the issue. That
>>>> modules and multitudes of features are not the issue. Previous efforts are
>>>> not the issue. Object oriented development and agile frameworks are not the
>>>> issue. And yet all remain pertinent software development conversation. Bike
>>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>>> them.
>>>>
>>>>
>>>> But discuss colors with someone else please. I can’t do it anymore.  But
>>>> I can discuss some jungle hack and slash momentum and trail blazing.  Give
>>>> me a moment. Time is fairly irrelevant on this dance floor. Tomorrow. Next
>>>> year. 3 years. Solr 42. It’s all the same timeline when the time comes.
>>>> Hell, my brain feels equally everywhere in all the timelines anyway. So one
>>>> moment, and I’ll point you to my speech, my journals, my old jungle dune
>>>> buggy - I did spend a moment or two once or twice preparing the damn thing
>>>> - just a moment or two, just 1, 2 or 3 times - and then the microphone is
>>>> all yours. But please, don’t talk to me about rainbows or shades of
>>>> magenta, I’m a jungle explorer, not a Sherwin Williams employee. Don’t
>>>> cover me with paint - I’m a roto spinning paint spewing machine. And I’d
>>>> love to spare you the paint ball war as well.
>>>>
>>>>
>>>> Links coming.
>>>>
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://about.me/markrmiller
>>>>
>>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
>

-- 
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have a
defensive posture towards the state of the world, preconceived biases
against me - let's try to trip the light fantastic and hard crash this
thing.The slow roll I'm wrapping up can fill in.

I'm gonna link bombs and jet around a bit.

Let's kind of root of this at "shards with no leader, replicas not
recovering, non functional Overseer etc"

You can't really get anywhere without that at least workable. You can't get
to scale and performance and understandability and many necessary
improvements without it at least semi solid. You can't obtain reliability
and greatness without it being ridiculously solid. To test and verify and
explore that, you'll have quite a challenge avoiding joining my contest. So
it's a good lead. But also, it's an undercurrent through the system. The
connections and loop backs and dependencies and cycles - you can try to say
what about this or that, but in the end we end up talking about the same
thing. The core behavior of the whole thing, that underpins and connects
everything. That can be addressed in many different ways, keeping things,
removing things, changing things - you could do it many ways. "The overseer
is not the problem", it must sound funny to you. But the Overseer as a
concept has no problem. Even the poor current design can essentially work.
30 designs can work better, other designs can work too. The root issues
around it don't stop at its borders. Anyway, let's do it. I'll dump the
path here. Let's try to blast a litte.







>
On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com> wrote:
>
Do you consider performance/scale of SolrCloud or corectness (shards with
>> no leader, replicas not recovering, non functional Overseer etc) as the
>> most important areas for improvement if we were to move to a v2?
>>
>>
>> Ilan
>>
>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com> a
>> écrit :
>>
>>> Now a I was saying ... SolrCloud2. Well, some people don’t like the name
>>> - fair enough - I don’t like purple much and I didn’t invent either... so
>>> Solr2.
>>>
>>>
>>>
>>> I have some thoughts, a few things to offer. Yes, the guy behind a lot
>>> of the current garbage. They asked me my thoughts on this thing on day one
>>> - I told them I don’t have a single one, let me have a go and I’ll get back
>>> to you. Done and done.
>>>
>>>
>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here - if I
>>> could have the floor for just a moment. I’ve put just a little bit of time
>>> and effort in before grabbing the mic. A modicum. My ego, my job, my
>>> status, my history, really not part of the equation, so if I could request
>>> a few lines before the rebuttal and correction and redirection.  No
>>> hurry for those things, they are patient characters.
>>>
>>>
>>> Anyway, give me a moment, and in return I’ll spare you the long winded
>>> diversions.
>>>
>>>
>>> Having considered Solr2 for some time, I see all kinds of roadblocks and
>>> restraints and limitations. I see a path fraught with the potential to
>>> mimic previous paths. And my goodness, I’m getting old. Fresh, promising
>>> paths please. So Ive invested some time and effort to establish an escape
>>> route from the safe, conservative hack and slash through the jungle that
>>> gets less safe the deeper we dive. I’d simply tell you about it, but jungle
>>> stories are all heart and no soul and we will all trade them all day.
>>>
>>>
>>> So I’ve got some notes and code and maps and journals and crap instead.
>>> Compiled from a couple expeditions. From before spelunking made me an old
>>> man. So my machete isn’t solo diving any new trails, single handedly
>>> slaying jungle cats any longer. The blade just keeps getting duller.
>>>
>>>
>>> But I’ve got these previous materials. I’ll lay them out. Take a look.
>>> If we can explore and discuss with just a passing courtesy of respect for
>>> our relative time and investment and focus put in before letting loose, I’m
>>> sure we can move beyond basic software counter meandering quickly enough to
>>> actually enter the jungle.
>>>
>>>
>>> I’ll try and lay out some evidence to help here. For instance, that
>>> zookeeper is not the issue. That the overseer is not the issue. That
>>> modules and multitudes of features are not the issue. Previous efforts are
>>> not the issue. Object oriented development and agile frameworks are not the
>>> issue. And yet all remain pertinent software development conversation. Bike
>>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>>> them.
>>>
>>>
>>> But discuss colors with someone else please. I can’t do it anymore.  But
>>> I can discuss some jungle hack and slash momentum and trail blazing.  Give
>>> me a moment. Time is fairly irrelevant on this dance floor. Tomorrow. Next
>>> year. 3 years. Solr 42. It’s all the same timeline when the time comes.
>>> Hell, my brain feels equally everywhere in all the timelines anyway. So one
>>> moment, and I’ll point you to my speech, my journals, my old jungle dune
>>> buggy - I did spend a moment or two once or twice preparing the damn thing
>>> - just a moment or two, just 1, 2 or 3 times - and then the microphone is
>>> all yours. But please, don’t talk to me about rainbows or shades of
>>> magenta, I’m a jungle explorer, not a Sherwin Williams employee. Don’t
>>> cover me with paint - I’m a roto spinning paint spewing machine. And I’d
>>> love to spare you the paint ball war as well.
>>>
>>>
>>> Links coming.
>>>
>>>
>>> Mark
>>>
>>>
>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>> --
> - Mark
>
> http://about.me/markrmiller
>

Re: Solr2

[Mark Miller <ma...@gmail.com>]

That sounds like a simple question. But for me, it is not.

I’m going to be frank. People love to take offense when I am frank. But
let’s be real, let’s discuss.

In both of those areas we are near the bottom of the totem pole. Many
others are as well and lower. Large numbers. But that does not change the
size of the totem pole or where we are on it.

And so, do I think this is important or that? Do I care about this or that?
Hmmm...

I’d like to climb the totem pole a bit.

And this is where people take offense. I’m sorry, children out of the room
please.

It’s actually a bear to climb. Lifting a city. The work I have done, will
blow this away. Now, it’s a jungle buggy, not your next jet fighter. And
I’m like, just in the early parts of the jungle. This buggy is still te
clearing bush ive already cleared.

People that are actually serious, let’s look at that and discuss before
getting offended.

I had to take the long route and try and catch my own hail marry because
the state of the world when I discovered most most of what I learned - good
god, I’m hardly into the brush, need to call in the military, and over here
I see a bunch of independent militias skirmishing. Some of the best
soldiers, checked out in the sickbay. Opposing soldiers seemingly hanging.
Non military refs running around. And that’s all fine. Your lives your
business. Don’t care. But need to call in a hit squad, not a ... whatever.

What we have is a kind of pretend parallel, sequence machine. Full of traps
and pitfalls, and don’t move that rock and Jesus, clear too much jungle and
watch out for the critters.

Again, fine. This type of software had a golden age. Happy for it. But this
is what I’m facing. I need serious collaborators. Not bitter frenemies.

Now we have benefits from the jungle. You basically get to ignore any real
software rules or laws or patterns or ... whatever. Glad you went to
college, you will have  fun applying that to a class or three every now and
then, but kick back, cause you don’t need it.

And it’s true. You don’t. The system compensates. To a degree. But do you
know at what expense? I’ll put huge good money pretty much no one working
on this does. Again. Not judgement. You know how much it took me to dig
out? To explore? To confirm? To experiment? If you have the time or effort
to do anything but the kind of intro and sneer that does often happen you
are a strange human or a goddamn genius above the ones  know.  The expense
is huge. And advancing hardware and pretend parallel and a flurry of hail
and did I miss? “No worries, hail being spammed everywhere every minute, it
will hit you” will compensate. Horrible blocking? Dangerous patterns?
Ridiculous over the top violations of rules that mattered before Java made
intel inside? So again, don’t think I’m saying this is shit software and a
caring way - I’m not saying that. It’s the totem pole.  This type of
software gets to survive and thrive by watching the pole grow and chilling
near the bottom.

And so. Make it work correct? Or make it work fast?

Well now it sounds silly. Is that a trick question?

Well I think it might be. For one, the two are related at a reasonable
level. And for two, how do you start building human structures vs jungle?
Tough question. Lots of people have answers and stuff that works all over.
But for this I unique situation and software here?

I suppose lots of opinions. Personally, I’ve tried to tackle it before.

Can you tackle it without a collection creation contest? Without pounding
cores and basic behavior and setup and shutdown and all of it at thousands
and thousands of everything in a ridiculous short time?

Yes. You could. It takes major tools and infrastructure and focus on none
feature shit. Like the developer support side is as critical as the user
production side.

Now I had and have to invest huge there myself. Takes way more than I could
handle or maintain and blah blah. I have a personal, huge, jungle scrap kit
of tools to figure out what the hell this things does and then what does
when I change it. And when I pound on it and create tremendous data that I
need to analyze and iterate on at a high right. You not only need the
tools, they have to have quick iteration. They have to. This jungle is
great at playing dead against simple, non agresiva tactics. But out in the
wild? Doesn’t work that way.

Anyway, without the reserve coming in to setup quite an infra investment,
you can cheat the hell out of the problem by putting the system into a
position that you can pound on it in massive parallel, and putting yourself
in the position of being able to absorb and respond with quick iteration.
Big hack when you have 0 coast guard coming, but also still likely
necessary work a small team. If you can’t break the rust in the wheels and
then stress this thing, it’s true behavior possibilities will be
outsmarting and outmaneuvering like any good jungle cat.

Now, with what I already learned from such efforts before, could I address
much more quicker?

Oh god yess. This thing is silly. Silly. The only hard thing about that, is
that it’s a bitch. Go shift some rocks in a true advancement and your going
to be stuck with dealing with gods knows what coming out of 5 other rocks
that get hit and some insects you disturbed with the air momentum.

There is a reason this level advanced happens ridiculously rarely even
though it’s huge plentiful and ridiculously obvious and non brilliant moves
when there are so many intelligent people involved, that have come and
gone.  Jungle don’t care how smart you are. Did you bring your X-ray
vision? Super speed? Backup hero’s?

Now, it also depends on your idea of correctness.
What you need. What you want. Those issues?  Garbage. It’s just garbage
going on. You can slow step through it and see. And just clear cut easy.

And then realize Jesus, what are we doing anyway. Those issues are not even
my concern. Unless you are talking at crazy scale and performance. Even
then, not the biggest issues at all. Part of the most basic of tables
stakes to even talk about the stuff we really want to do. When I say this
thing is so forkable, it’s not a threat at all. It’s observation that in my
world. That wouldn’t be a question. If immediately move to level 5 or 7
just to start and that’s level 1.

It’s embarrassing that stuff is the problem it is. You can make it - no
exaggerating one tiny bit - 10x better, and it’s still nothing you should
turn in for a grade. And I’m being emotionally honest. Like let’s talk for
real. It doesn’t change how the software works and is working -  how many
people are using it - the value it’s given and giving. Don’t make me lie or
hide legitimate and non biased honesty. I don’t pick it. I feel bad if it
hurts or offends or bothers anyone. That’s affected me for a long time in
terms of keeping me in the dark. I don’t want to offend anyone. But don’t
make me the villain  because i refuse to not point at the elephant anymore.
I’m not the villain. I’d prefer you think I’m crazy.  Where people have a
misconception is that I care about whew ethos specific project is. God no.
With my old Lucene buddies it would be great. Trench war fair. We would buy
beat and wage war. But they ain’t here. And so no, I don’t care where the
software is. I care about where it cold and should be and what we can do.
What the potential. I go, oh bay, look at the room and potential in that
totem pole! That shit looks good to me. Ground to move. Anyone offended or
in disagreement, no problem, look the short side   That’s fine. Different
strokes.

I’m gonna and talk about some terrible behavior and easy improvements. With
a smile on my face, not a frown at the state of things.





On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com> wrote:

> Do you consider performance/scale of SolrCloud or corectness (shards with
> no leader, replicas not recovering, non functional Overseer etc) as the
> most important areas for improvement if we were to move to a v2?
>
>
> Ilan
>
> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com> a
> écrit :
>
>> Now a I was saying ... SolrCloud2. Well, some people don’t like the name
>> - fair enough - I don’t like purple much and I didn’t invent either... so
>> Solr2.
>>
>>
>>
>> I have some thoughts, a few things to offer. Yes, the guy behind a lot of
>> the current garbage. They asked me my thoughts on this thing on day one - I
>> told them I don’t have a single one, let me have a go and I’ll get back to
>> you. Done and done.
>>
>>
>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here - if I
>> could have the floor for just a moment. I’ve put just a little bit of time
>> and effort in before grabbing the mic. A modicum. My ego, my job, my
>> status, my history, really not part of the equation, so if I could request
>> a few lines before the rebuttal and correction and redirection.  No
>> hurry for those things, they are patient characters.
>>
>>
>> Anyway, give me a moment, and in return I’ll spare you the long winded
>> diversions.
>>
>>
>> Having considered Solr2 for some time, I see all kinds of roadblocks and
>> restraints and limitations. I see a path fraught with the potential to
>> mimic previous paths. And my goodness, I’m getting old. Fresh, promising
>> paths please. So Ive invested some time and effort to establish an escape
>> route from the safe, conservative hack and slash through the jungle that
>> gets less safe the deeper we dive. I’d simply tell you about it, but jungle
>> stories are all heart and no soul and we will all trade them all day.
>>
>>
>> So I’ve got some notes and code and maps and journals and crap instead.
>> Compiled from a couple expeditions. From before spelunking made me an old
>> man. So my machete isn’t solo diving any new trails, single handedly
>> slaying jungle cats any longer. The blade just keeps getting duller.
>>
>>
>> But I’ve got these previous materials. I’ll lay them out. Take a look. If
>> we can explore and discuss with just a passing courtesy of respect for our
>> relative time and investment and focus put in before letting loose, I’m
>> sure we can move beyond basic software counter meandering quickly enough to
>> actually enter the jungle.
>>
>>
>> I’ll try and lay out some evidence to help here. For instance, that
>> zookeeper is not the issue. That the overseer is not the issue. That
>> modules and multitudes of features are not the issue. Previous efforts are
>> not the issue. Object oriented development and agile frameworks are not the
>> issue. And yet all remain pertinent software development conversation. Bike
>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>> them.
>>
>>
>> But discuss colors with someone else please. I can’t do it anymore.  But
>> I can discuss some jungle hack and slash momentum and trail blazing.  Give
>> me a moment. Time is fairly irrelevant on this dance floor. Tomorrow. Next
>> year. 3 years. Solr 42. It’s all the same timeline when the time comes.
>> Hell, my brain feels equally everywhere in all the timelines anyway. So one
>> moment, and I’ll point you to my speech, my journals, my old jungle dune
>> buggy - I did spend a moment or two once or twice preparing the damn thing
>> - just a moment or two, just 1, 2 or 3 times - and then the microphone is
>> all yours. But please, don’t talk to me about rainbows or shades of
>> magenta, I’m a jungle explorer, not a Sherwin Williams employee. Don’t
>> cover me with paint - I’m a roto spinning paint spewing machine. And I’d
>> love to spare you the paint ball war as well.
>>
>>
>> Links coming.
>>
>>
>> Mark
>>
>>
>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
- Mark

http://about.me/markrmiller

Re: Solr2

[Mark Miller <ma...@gmail.com>]

Anyway, that’s just a bit of context.  What and how and when a Solr 2 could
be and work and coexist and share with what we have is hugely available and
flexible issue - if I’m talking about o serious collaborators. Almost as
flexible and open to possibility as software itself.

And I’ve got plenty coming very very soon that’s organized and approachable
and picky eater edible.  So no reason to get bogged down first.

But also id like to say, that question is not anywhere near the trade off
you think it is. Try to avoid huge gains in both with simple potent
strikes. Not the issue. Until I have to resort to huge, aggressive scale
and performance, no thinking about that trade.  Both will rise on anything
good. Let’s discuss that a little. I’ll put something together.

Beyond that - and there is work and time between us and beyond that - but
beyond that, is where the interesting field lies. The point that hardware
and inefficiency gets routed. Enemy
ambushes deep in victory territory?!

Now we can author the book backwards.  You tell me.

Does that mean we don’t peak at the end? Discuss the end? hang the time in
the sqaure before he messed up the timeline?

Who knows.  But that is where this project needs to move too. And more
offensive and laughter - but in limited time. Because projects are being
built working on those problems while projects like ours can’t even look at
them yet.  I’m not taking a keep when I say time is limited. It’s just
obvious nature as you peel around at the state of things.



On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <il...@gmail.com> wrote:

> Do you consider performance/scale of SolrCloud or corectness (shards with
> no leader, replicas not recovering, non functional Overseer etc) as the
> most important areas for improvement if we were to move to a v2?
>
>
> Ilan
>
> Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com> a
> écrit :
>
>> Now a I was saying ... SolrCloud2. Well, some people don’t like the name
>> - fair enough - I don’t like purple much and I didn’t invent either... so
>> Solr2.
>>
>>
>>
>> I have some thoughts, a few things to offer. Yes, the guy behind a lot of
>> the current garbage. They asked me my thoughts on this thing on day one - I
>> told them I don’t have a single one, let me have a go and I’ll get back to
>> you. Done and done.
>>
>>
>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here - if I
>> could have the floor for just a moment. I’ve put just a little bit of time
>> and effort in before grabbing the mic. A modicum. My ego, my job, my
>> status, my history, really not part of the equation, so if I could request
>> a few lines before the rebuttal and correction and redirection.  No
>> hurry for those things, they are patient characters.
>>
>>
>> Anyway, give me a moment, and in return I’ll spare you the long winded
>> diversions.
>>
>>
>> Having considered Solr2 for some time, I see all kinds of roadblocks and
>> restraints and limitations. I see a path fraught with the potential to
>> mimic previous paths. And my goodness, I’m getting old. Fresh, promising
>> paths please. So Ive invested some time and effort to establish an escape
>> route from the safe, conservative hack and slash through the jungle that
>> gets less safe the deeper we dive. I’d simply tell you about it, but jungle
>> stories are all heart and no soul and we will all trade them all day.
>>
>>
>> So I’ve got some notes and code and maps and journals and crap instead.
>> Compiled from a couple expeditions. From before spelunking made me an old
>> man. So my machete isn’t solo diving any new trails, single handedly
>> slaying jungle cats any longer. The blade just keeps getting duller.
>>
>>
>> But I’ve got these previous materials. I’ll lay them out. Take a look. If
>> we can explore and discuss with just a passing courtesy of respect for our
>> relative time and investment and focus put in before letting loose, I’m
>> sure we can move beyond basic software counter meandering quickly enough to
>> actually enter the jungle.
>>
>>
>> I’ll try and lay out some evidence to help here. For instance, that
>> zookeeper is not the issue. That the overseer is not the issue. That
>> modules and multitudes of features are not the issue. Previous efforts are
>> not the issue. Object oriented development and agile frameworks are not the
>> issue. And yet all remain pertinent software development conversation. Bike
>> shed rainbows. You can make beautiful multicolored bike sheds with all of
>> them.
>>
>>
>> But discuss colors with someone else please. I can’t do it anymore.  But
>> I can discuss some jungle hack and slash momentum and trail blazing.  Give
>> me a moment. Time is fairly irrelevant on this dance floor. Tomorrow. Next
>> year. 3 years. Solr 42. It’s all the same timeline when the time comes.
>> Hell, my brain feels equally everywhere in all the timelines anyway. So one
>> moment, and I’ll point you to my speech, my journals, my old jungle dune
>> buggy - I did spend a moment or two once or twice preparing the damn thing
>> - just a moment or two, just 1, 2 or 3 times - and then the microphone is
>> all yours. But please, don’t talk to me about rainbows or shades of
>> magenta, I’m a jungle explorer, not a Sherwin Williams employee. Don’t
>> cover me with paint - I’m a roto spinning paint spewing machine. And I’d
>> love to spare you the paint ball war as well.
>>
>>
>> Links coming.
>>
>>
>> Mark
>>
>>
>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
- Mark

http://about.me/markrmiller

Re: Solr2

[Ilan Ginzburg <il...@gmail.com>]

Do you consider performance/scale of SolrCloud or corectness (shards with
no leader, replicas not recovering, non functional Overseer etc) as the
most important areas for improvement if we were to move to a v2?

Ilan

Le sam. 10 avr. 2021 à 22:37, Mark Miller <ma...@gmail.com> a écrit :

> Now a I was saying ... SolrCloud2. Well, some people don’t like the name -
> fair enough - I don’t like purple much and I didn’t invent either... so
> Solr2.
>
>
>
> I have some thoughts, a few things to offer. Yes, the guy behind a lot of
> the current garbage. They asked me my thoughts on this thing on day one - I
> told them I don’t have a single one, let me have a go and I’ll get back to
> you. Done and done.
>
>
> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here - if I
> could have the floor for just a moment. I’ve put just a little bit of time
> and effort in before grabbing the mic. A modicum. My ego, my job, my
> status, my history, really not part of the equation, so if I could request
> a few lines before the rebuttal and correction and redirection.  No hurry
> for those things, they are patient characters.
>
>
> Anyway, give me a moment, and in return I’ll spare you the long winded
> diversions.
>
>
> Having considered Solr2 for some time, I see all kinds of roadblocks and
> restraints and limitations. I see a path fraught with the potential to
> mimic previous paths. And my goodness, I’m getting old. Fresh, promising
> paths please. So Ive invested some time and effort to establish an escape
> route from the safe, conservative hack and slash through the jungle that
> gets less safe the deeper we dive. I’d simply tell you about it, but jungle
> stories are all heart and no soul and we will all trade them all day.
>
>
> So I’ve got some notes and code and maps and journals and crap instead.
> Compiled from a couple expeditions. From before spelunking made me an old
> man. So my machete isn’t solo diving any new trails, single handedly
> slaying jungle cats any longer. The blade just keeps getting duller.
>
>
> But I’ve got these previous materials. I’ll lay them out. Take a look. If
> we can explore and discuss with just a passing courtesy of respect for our
> relative time and investment and focus put in before letting loose, I’m
> sure we can move beyond basic software counter meandering quickly enough to
> actually enter the jungle.
>
>
> I’ll try and lay out some evidence to help here. For instance, that
> zookeeper is not the issue. That the overseer is not the issue. That
> modules and multitudes of features are not the issue. Previous efforts are
> not the issue. Object oriented development and agile frameworks are not the
> issue. And yet all remain pertinent software development conversation. Bike
> shed rainbows. You can make beautiful multicolored bike sheds with all of
> them.
>
>
> But discuss colors with someone else please. I can’t do it anymore.  But
> I can discuss some jungle hack and slash momentum and trail blazing.  Give
> me a moment. Time is fairly irrelevant on this dance floor. Tomorrow. Next
> year. 3 years. Solr 42. It’s all the same timeline when the time comes.
> Hell, my brain feels equally everywhere in all the timelines anyway. So one
> moment, and I’ll point you to my speech, my journals, my old jungle dune
> buggy - I did spend a moment or two once or twice preparing the damn thing
> - just a moment or two, just 1, 2 or 3 times - and then the microphone is
> all yours. But please, don’t talk to me about rainbows or shades of
> magenta, I’m a jungle explorer, not a Sherwin Williams employee. Don’t
> cover me with paint - I’m a roto spinning paint spewing machine. And I’d
> love to spare you the paint ball war as well.
>
>
> Links coming.
>
>
> Mark
>
>
>
> --
> - Mark
>
> http://about.me/markrmiller
>